Jekyll2021-02-25T19:37:02+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comWhy did Network to Code Fork NetBox?2021-02-25T00:00:00+00:002021-02-25T00:00:00+00:00https://blog.networktocode.com/post/why-did-network-to-code-fork-netbox<p><em>[This post was reviewed and edited by many of the people who have contributed to the Nautobot project at Network to Code.]</em></p> <p>Over the last five years, NetBox has gained adoption as an IP Address Management (IPAM) and Data Center Infrastructure Management (DCIM) platform while also serving as a Source of Truth for documenting networks and network automation projects. In fact, Network to Code has been heavy users, supporters, and contributors of NetBox for years. While we started offering NetBox professional services in 2018 and support for NetBox in 2019, we have continued to support an ever-growing customer base that has evolving requirements around Source of Truth and more broadly around creating holistic network automation solutions. Because of this, and in order to better serve our customers and the community, Network to Code made the difficult decision to fork NetBox and drive it in new directions.</p> <p>Let’s dive into the details.</p> <h2 id="background">Background</h2> <p>Network to Code has been a power user of NetBox for the past 3+ years. We’ve designed and deployed numerous solutions that revolve around and integrate with NetBox. We have used the existing models, extensibility features, worked on a myriad of private customer-specific forks, created ways to manage settings and URLs, as well as created plugins for various use cases. In short, over the course of countless projects, we have seen first-hand what works well for users and what has room for improvement.</p> <p>There have been questions raised in different public forums if Network to Code has been a contributor to NetBox. The answer is yes. Let’s explore some of those contributions.</p> <p>First and foremost, as part of the NetBox community, Network to Code has contributed <a href="https://www.google.com/search?source=hp&amp;ei=ttU3YLj0MNL99AOe2pqABQ&amp;iflsig=AINFCbYAAAAAYDfjxtBxUgzVc2PSH2M32BrOF3y9d_6U&amp;q=site%3Ablog.networktocode.com+netbox&amp;oq=site%3Ablog.networktocode.com+netbox&amp;gs_lcp=Cgdnd3Mtd2l6EAM6DgguELEDEMcBEKMCEJMCOgsILhCxAxDHARCjAjoICAAQsQMQgwE6BQgAELEDOggILhDHARCjAjoCCAA6BQguEJMCOg4ILhCxAxCDARDHARCvAToFCC4QsQM6CwguEMcBEKMCEJMCOgQIABAKULgMWItLYNxYaAFwAHgAgAFoiAHPEJIBBDIyLjKYAQCgAQGgAQKqAQdnd3Mtd2l6sAEA&amp;sclient=gws-wiz&amp;ved=0ahUKEwi4p5LGv4XvAhXSPn0KHR6tBlAQ4dUDCAk&amp;uact=5">numerous blog articles</a>, <a href="https://www.youtube.com/channel/UCwBh-dDdoqzxXKyvTw3BuTw/search?query=netbox">published videos</a>, <a href="https://www.eventbrite.com/e/netbox-day-presented-by-network-to-code-tickets-102186514616#">hosted a NetBox Day</a>, presented at conferences, enabled and participated in innumerable Slack discussions hosted on the <a href="https://slack.networktocode.com">Network to Code Slack workspace</a>, participated in podcasts, hosted low-cost training, and more. While all of this was not hand-to-keyboard development, NTC has undoubtedly contributed thousands of hours toward NetBox in this capacity in recent years.</p> <p>For contributions to the core of NetBox, NTC employees (not counting the lead maintainer of NetBox, who was employed by NTC for 18 months and spent the majority of his time actively maintaining NetBox during that time) have contributed 78 issues and 35 PRs, two of which are still being worked on. Some of these contributions have been significant in scope, such as the initial implementation of the NetBox plugin API that was released in NetBox 2.8.</p> <p>When you look at code and development, there are NTC projects that also build on top of NetBox. We have built a <a href="https://github.com/networktocode/ntc-netbox-plugin-onboarding">Device Onboarding plugin</a>, <a href="https://github.com/networktocode/ntc-netbox-plugin-metrics-ext">Prometheus metrics plugin</a>, a number of other plugins that we are working to open source, as well as projects like <a href="https://github.com/networktocode/network-importer">network-importer</a> to simplify adding data into NetBox. Each of these projects individually are the result of hundreds of hours, adding up to many months of dedicated development effort.</p> <p>While we have never claimed to be an official sponsor of NetBox, we were effectively an unofficial and informal sponsor of the project. Lastly, one of the core committers on our fork had also been a maintainer of NetBox for a number of years while at Network to Code.</p> <p>We have always had candid discussions on feature requests, support, and on the direction that we believed was in the best interest of the long term success of NetBox. As time went on and after numerous discussions, there became a growing divergence in our vision about what a Source of Truth for networking should look like and how to get there, and the NetBox project team suggested that we should consider forking. Late in 2020, we took a step back to think about the options.</p> <p>Taking what we’ve learned deploying and integrating NetBox solutions for customers over the past few years, what we have seen and heard from the community, coupled with discussions with the NetBox project team, Network to Code came to the conclusions that it was best to fork NetBox with the vision and direction necessary for powering network automation platforms. Next, from a business and technical perspective, let’s explore the key reasons we considered and what drove us to make the decision.</p> <h3 id="we-need-to-provide-world-class-support-for-our-clients">We need to provide world class support for our clients.</h3> <p>We need to offer high-touch support models with Service Level Agreements (SLAs) we can guarantee. We need flexibility to offer Long-term Support (LTS) for customers who can’t upgrade at the pace of a fast moving open source project. In addition, we need to be able to deliver the features required to get a system into production in an enterprise environment. Some examples include simplifying the ability to integrate with Single Sign-On (SSO) services such as SAML or OIDC and adding support for additional database backends such as MySQL.</p> <h3 id="define-a-strategic-vision-for-source-of-truth-that-enables-network-automation">Define a strategic vision for Source of Truth that enables network automation.</h3> <p>We firmly believe that a Source of Truth is a fundamental requirement to enable data-driven network automation. This is where we are able to leverage our collective experience at Network to Code and add functionality that simplifies integrating with network automation solutions. While the new platform can still be used for network documentation, its key focus is on enabling network automation. After all, network automation is our core business at Network to Code. It has become apparent through our customer projects that there is a need for more flexible APIs and more seamless ways to integrate with other data sources.</p> <h3 id="build-a-network-automation-app-platform">Build a Network Automation App Platform.</h3> <p>This is one area that we think is quite unique. The goal is to leverage the rich data stored in the Source of Truth and build high-value apps that use and consume that data while interacting with other systems and network devices as needed. The best metaphor to consider is a smartphone. It will always be a “phone.” However, this “phone” is a delivery mechanism for high-value apps that we all get to use on a daily basis (should you choose to). In this example, the Source of Truth is the phone, i.e. the delivery mechanism. Our goal is to enable the Source of Truth as a platform through which high-value apps are deployed. These apps can be lightweight (creating new models and APIs) or be used to deliver solutions (pre/post change, integrate with monitoring, perform backups). We believe it should be up to the users to decide.</p> <h2 id="community-committed">Community Committed</h2> <p>Network to Code has deep roots in the community. As mentioned earlier, we have spent thousands of hours on community work around NetBox. If you look at every other project we’ve been a part of, that number goes up significantly. From open sourcing various projects of our own to contributing to other such projects to engaging in discussions on numerous forums, community is extremely important to who we are as a company.</p> <p>We know there is A LOT of work to do to gain the respect and trust of the community as we move forward with Nautobot. We are committed to fostering user and developer communities that are fully transparent. We will be hanging out in the #nautobot channel on the <a href="slack.networktocode.com">Network to Code Slack</a>. Please come join and let us know what you think! You can also view the <a href="https://www.networktocode.com/nautobot/roadmap">initial Nautobot roadmap</a>.</p> <p>If you have ideas, enhancements, or feature requests for Nautobot, please don’t hesitate to open an <a href="https://github.com/nautobot/nautobot/issues">issue</a> on GitHub.</p> <h2 id="going-forward">Going Forward</h2> <p>Just to get us where we are with Nautobot, we have invested close to a “work year” of effort across our team just over the past few months. This effort and focus is not going to stop. While Nautobot is much more than NetBox with additional features, it’s actually not even about what you see today. This is just the first release. If you haven’t already done so, please check out the <a href="https://www.networktocode.com/nautobot/roadmap">roadmap</a>. You’ll see that our goal is to empower flexibility and extensibility while truly being a Source of Truth foundation for network automation. Our vision will see Nautobot continue to diverge from NetBox in such a way that the separation of the two will become much more distinct.</p> <h3 id="git-history">Git History</h3> <p>When we first released the Nautobot project on GitHub, we had made a decision, based on technical considerations, to not keep the complete NetBox commit history. It was never our intent to “erase” the NetBox commit history because that history exists within the NetBox project and we are making every attempt to be fully transparent about the forked status and attribute proper credit. We have heard your feedback that this can give the appearance of dismissing the history and community engagement that has brought NetBox to where it is today, and we apologize for that. We are grateful and we are re-adding the NetBox Git history to Nautobot. Below are the technical reasons we had in mind when we chose to reset the commit history.</p> <p>First, we thought that it would make it possible to draw a clear dividing line between the development of NetBox and the development of Nautobot so there would be no ambiguity as to whether a given commit in the Git history is a shared NetBox commit that was inherited by Nautobot, or whether a commit is specific to Nautobot and distinct from NetBox. This would make it easier to see over time how the two projects have diverged.</p> <p>Second, we wanted to avoid any ambiguity about version numbers. Creating a new repository from our own version 1.0 allowed us to avoid these collisions and confusion without needing to do anything messy like deleting old tags and releases from the history.</p> <p>Third, we have made a large number of invasive changes related to packaging and renaming which greatly reduce the historical relevance of the history from a purely technical standpoint.</p> <p>Fourth, it reduced the size of the repository substantially. A fresh clone of <code class="highlighter-rouge">nautobot.git</code> is only about 28 MB in size, versus a fresh clone of upstream <code class="highlighter-rouge">netbox.git</code> weighing in at about 185 MB.</p> <p>We remain grateful to all developers of and contributors to NetBox, and hope that this provides appropriate clarity as to the reasons why we initially chose this particular approach.</p> <h2 id="closing">Closing</h2> <p>The decision to fork and create something new was not a decision we took lightly at Network to Code. We are committed to creating something great and to ensuring Nautobot will become a thriving, transparent, and engaging project and community. We’re already seeing a lot of positive feedback from the community based on the <a href="https://www.networktocode.com/nautobot/roadmap">direction</a> planned for Nautobot.</p> <p>We welcome you on the journey and hope you come along for the ride. We look forward to the evolution of Nautobot and seeing Nautobot used to power network automation solutions across the world.</p> <p>-Jason</p>Jason Edelman[This post was reviewed and edited by many of the people who have contributed to the Nautobot project at Network to Code.]Introducing Schema Enforcer2021-02-11T00:00:00+00:002021-02-11T00:00:00+00:00https://blog.networktocode.com/post/introducing_schema_enforcer<p>These days, most organizations heavily leverage YAML and JSON to store and organize all sorts of data. This is done in order to define variables, be provided as input for generating device configurations, define inventory, and for many other use cases. Both YAML and JSON are very popular because both languages are very flexible and are easy to use. It is relatively easy for users who have little to no experience working with structured data (as well as for very experienced programmers) to use JSON and YAML because the formats do not require users to define a schema in order to define data.</p> <p>As the use of structured data increases, the flexibility provided because these languages don’t require data to adhere to a schema create complexity and risk. If a user accidentally defines the data for <code class="highlighter-rouge">ntp_servers</code> in two different structures (e.g. one is a list, and one is a dictionary), automation tooling must be written to handle the differences in inputs in some way. Often times, the automation tooling just bombs out with a cryptic message in such cases. This is because the tool consuming this data rightfully expects to have a contract with it, that the data will adhere to a clearly defined form and thus the tool can interact with the data in a standard way. It is for this reason that APIs, when updated, should never change the format in which they provide data unless there is some way to delineate the new format (e.g. an API version increment). By ensuring data is defined in a standard way, complexity and risk can be mitigated.</p> <p>With structured data languages like YAML and JSON which do not inherently define a schema (contract) for the data they define, a schema definition language can be used to provide this contract, thereby mitigating complexity and risk. Schema definition languages come with their own added maintenance though as the burden of writing the logic to ensure structured data is schema valid falls on the user. The user doesn’t just need to maintain structured data and schemas, they also have to build and maintain the tooling that checks if data is schema valid. To allow users to simply write schemas and structured data and worry less about writing and maintaining the code that bolts them together, Network to Code has developed a tool called Schema Enforcer. Today we are happy to announce that we are making Schema Enforcer available to the community.</p> <blockquote> <p>Check it out on <a href="https://github.com/networktocode/schema-enforcer/">Github</a>!</p> </blockquote> <h2 id="what-is-schema-enforcer">What is Schema Enforcer</h2> <p>Schema Enforcer is a framework for allowing users to define schemas for their structured data and assert that the defined structured data adheres to their schemas. This structured data can (currently) come in the the form of a data file in JSON or YAML format, or an Ansible inventory. The schema definition is defined by using the <a href="https://json-schema.org/">JSONSchema</a> language in YAML or JSON format.</p> <h2 id="why-use-schema-enforcer">Why use Schema Enforcer?</h2> <p>If you’re familiar with JSONSchema already, you may be thinking “wait, doesn’t JSONSchema do all of this?”. JSONSchema does allow you to validate that structured data adheres to a schema definition, but it requires for you to write your own code to interact with and manage the data’s adherence to defined schema. Schema Enforcer is meant to provide a wrapper which makes it easy for users to manage structured data without needing to write their own code to check their structured data for adherence to a schema. It provides the following advantages over <em>just</em> using JSONSchema:</p> <ul> <li>Provides a framework for mapping data files to the schema definitions against which they should be checked for adherence</li> <li>Provides a framework for validating that Ansible inventory adheres to a schema definition or multiple schema definitions</li> <li>Prints clear log messages indicating each data object examined which is not adherent to schema, and the specific way in which these data objects are not adherent</li> <li>Allows a user to define unit tests asserting that their schema definitions are written correctly (e.g. that non-adherent data fails validation in a specific way, and adherent data passes validation)</li> <li>Exits with an exit code of <code class="highlighter-rouge">1</code> in the event that data is not adherent to schema. This makes it fit for use in a CI pipeline along-side linters and unit tests</li> </ul> <h2 id="an-example">An Example</h2> <p>I’ve created the following directories and files in a repository called <code class="highlighter-rouge">new_example</code>.</p> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-intro2.png" alt="Schema Enforcer Intro 2" /></p> <p>The directory includes</p> <ul> <li>structured data (in YAML format) defining ntp servers for the host <code class="highlighter-rouge">chi-beijing-rt01</code> inside of the file at <code class="highlighter-rouge">hostvars/chi-beijing-rt01/ntp.yml</code></li> <li>a schema definition inside of the file at <code class="highlighter-rouge">schema/schemas/ntp.yml</code></li> </ul> <p>If we examine the file at <code class="highlighter-rouge">hostvars/chi-being-rt01/ntp.yml</code> we can see the following data defined in YAML format.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># jsonschema: schemas/ntp</span> <span class="nn">---</span> <span class="na">ntp_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s">192.2.0.1</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s">192.2.0.2</span> </code></pre></div></div> <p>Note the comment <code class="highlighter-rouge"># jsonschema: schemas/ntp</code> at the top of the YAML file. This comment is used to declare the schema that the data in this file should be checked for adherence to, as well as the language being used to define the schema (JSONSchema here). Multiple schemas can be declared by comma separating them in the comment. For instance, the comment <code class="highlighter-rouge"># jsonschema: schemas/ntp,schemas/syslog</code> would declare that the data should be checked for adherence to two schema, one schema with the ID <code class="highlighter-rouge">schemas/ntp</code> and another with the id <code class="highlighter-rouge">schemas/syslog</code>. We can validate that this mapping is being inferred correctly by running the command <code class="highlighter-rouge">schema-enforcer validate --show-checks</code></p> <blockquote> <p>The <code class="highlighter-rouge">--show-checks</code> flag shows each data file along with a list of every schema IDs it will be checked for adherence to.</p> </blockquote> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-intro3.png" alt="Schema Enforcer Intro 3" /></p> <blockquote> <p>Other mechanisms for mapping data files to schemas against which they should be validated. See docs/mapping_schemas.md in the Schema Enforcer git repository. for more details. <br /> YAML supports the addition of comments using an octothorp. JSON does not support the addition of comments. To this end, only data defined in YAML format can declare the schema to which it should adhere with a comment. Another mechanism for mapping needs to be used if your data is defined in JSON format.</p> </blockquote> <p>If we examine the file at <code class="highlighter-rouge">schema/schemas/ntp.yml</code> we can see the following schema definition. This is written in the JSONSchema language and formatted in YAML.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="s">$schema</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://json-schema.org/draft-07/schema#"</span> <span class="s">$id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">schemas/ntp"</span> <span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">NTP</span><span class="nv"> </span><span class="s">Configuration</span><span class="nv"> </span><span class="s">schema."</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">ntp_servers</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">array"</span> <span class="na">items</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">address</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">format</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ipv4"</span> <span class="na">vrf</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">address"</span> <span class="na">uniqueItems</span><span class="pi">:</span> <span class="no">true</span> <span class="na">additionalProperties</span><span class="pi">:</span> <span class="no">false</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">ntp_servers"</span> </code></pre></div></div> <p>The schema definition above is used to ensure that:</p> <ul> <li>The <code class="highlighter-rouge">ntp_servers</code> property is of type hash/dictionary (object in JSONSchema parlance)</li> <li>No top level keys can be defined in the data file besides <code class="highlighter-rouge">ntp_servers</code></li> <li>It’s value is of type array/list</li> <li>Each item in this array must be unique</li> <li>Each element of this array/list is a dictionary with the possible keys <code class="highlighter-rouge">name</code>, <code class="highlighter-rouge">address</code> and <code class="highlighter-rouge">vrf</code> <ul> <li>Of these keys, <code class="highlighter-rouge">address</code> is required, <code class="highlighter-rouge">name</code> and <code class="highlighter-rouge">vrf</code> can optionally be defined, but it is not necessary to define them.</li> <li><code class="highlighter-rouge">address</code> must be of type “string” and it must be a valid IP address</li> <li><code class="highlighter-rouge">name</code> must be of type “string” if it is defined</li> <li><code class="highlighter-rouge">vrf</code> must be of type “string” if it is defined</li> </ul> </li> </ul> <p>Here is an example of the structured data being checked for adherence to the schema definition.</p> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-intro1.png" alt="Schema Enforcer Intro 1" /></p> <p>We can see that when <code class="highlighter-rouge">schema-enforcer</code> runs, it shows that all files containing structured data are schema valid. Also note that Schema Enforcer exits with a code of 0.</p> <p>What happens if we modify the data such that the first ntp server defined has a value of the boolean <code class="highlighter-rouge">true</code> and add a <code class="highlighter-rouge">syslog_servers</code> dictionary/hash type object at the top level of the YAML file.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># jsonschema: schemas/ntp</span> <span class="nn">---</span> <span class="na">ntp_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s">192.2.0.2</span> <span class="na">syslog_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s">192.0.5.3</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-intro4.png" alt="Schema Enforcer Intro 4" /></p> <p>We can see that two errors are flagged. The first informs us that the first element in the array which is the value of the <code class="highlighter-rouge">ntp_servers</code> top level key is a boolean and a string was expected. The second informs us that the additional top level property <code class="highlighter-rouge">syslog_servers</code> is a property that is additional to (is not specified in) the properties defined by the schema, and that additional properties are not allowed per the schema definition. Note that <code class="highlighter-rouge">schema-enforcer</code> exits with a code of 1 indicating a failure. If Schema Enforcer were to be used before structured data is ingested into automation tools as part of a pipeline, the pipeline would never have the automation tools consume the malformed data.</p> <h2 id="validating-ansible-inventory">Validating Ansible Inventory</h2> <p>Schema Enforcer supports validating that variables defined in an Ansible inventory adhere to a schema definition (or multiple schema definitions).</p> <p>To do this, Schema Enforcer first constructs a dictionary containing key/value pairs for each attribute defined in the inventory. It does this by flattening the varibles from the groups the host is a part of. After doing this, <code class="highlighter-rouge">schema-enforcer</code> maps which schemas it should use to validate the hosts variables in one of two ways:</p> <ul> <li>By using a list of schema ids defined by the <code class="highlighter-rouge">schema_enforcer_schema_ids</code> attribute (defined at the host or group level).</li> <li>By automatically mapping a schema’s top level properties to the Ansible host’s keys.</li> </ul> <p>That may have been gibberish on first pass, but the examples in the following sections will hopefully make things clearer.</p> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-ansible.png" alt="Schema Enforcer Ansible 1" /></p> <h2 id="an-examle-of-validating-ansible-variables">An Examle of Validating Ansible Variables</h2> <p>In the following example, we have an inventory file which defines three groups, <code class="highlighter-rouge">nyc</code>, <code class="highlighter-rouge">spine</code>, and <code class="highlighter-rouge">leaf</code>. <code class="highlighter-rouge">spine</code> and <code class="highlighter-rouge">leaf</code> are children of <code class="highlighter-rouge">nyc</code>.</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[nyc:children]</span> <span class="err">spine</span> <span class="err">leaf</span> <span class="nn">[spine]</span> <span class="err">spine1</span> <span class="err">spine2</span> <span class="nn">[leaf]</span> <span class="err">leaf1</span> <span class="err">leaf2</span> </code></pre></div></div> <p>The group <code class="highlighter-rouge">spine.yaml</code> has two top level keys; <code class="highlighter-rouge">dns_servers</code> and <code class="highlighter-rouge">interfaces</code>.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">cat group_vars/spine.yaml</span> <span class="nn">---</span> <span class="na">dns_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.2.2.2"</span> <span class="na">interfaces</span><span class="pi">:</span> <span class="na">swp1</span><span class="pi">:</span> <span class="na">role</span><span class="pi">:</span> <span class="s2">"</span><span class="s">uplink"</span> <span class="na">swp2</span><span class="pi">:</span> <span class="na">role</span><span class="pi">:</span> <span class="s2">"</span><span class="s">uplink"</span> <span class="na">schema_enforcer_schema_ids</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">schemas/dns_servers"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">schemas/interfaces"</span> </code></pre></div></div> <p>Note the <code class="highlighter-rouge">schema_enforcer_schema_ids</code> variable. This declaratively tells Schema Enforcer which schemas to use when running tests to ensure that the Ansible host vars for every host in the <code class="highlighter-rouge">spine</code> group are schema valid.</p> <p>Here is the interfaces schema which is declared above:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">bash$ cat schema/schemas/interfaces.yml</span> <span class="nn">---</span> <span class="s">$schema</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://json-schema.org/draft-07/schema#"</span> <span class="s">$id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">schemas/interfaces"</span> <span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Interfaces</span><span class="nv"> </span><span class="s">configuration</span><span class="nv"> </span><span class="s">schema."</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">interfaces</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">patternProperties</span><span class="pi">:</span> <span class="s">^swp.*$</span><span class="pi">:</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">description</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">role</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> </code></pre></div></div> <p>Note that the <code class="highlighter-rouge">$id</code> property is what is being declared by the <code class="highlighter-rouge">schema_enforcer_schema_ids</code> variable.</p> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-ansible2.png" alt="Schema Enforcer Ansible 2" /></p> <p>When we run the <code class="highlighter-rouge">schema-enforcer ansible</code> command with the <code class="highlighter-rouge">--show-pass</code> flag, we can see that the <code class="highlighter-rouge">spine1</code> and <code class="highlighter-rouge">spine2</code>’s defined <code class="highlighter-rouge">dns_servers</code> attribute did not adhere to schema.</p> <blockquote> <p>By default, <code class="highlighter-rouge">schema-enforcer</code> prints a “FAIL” message to stdout for each object in the data file which does not adhere to schema. If no objects fail to adhere to schema definitions, a single line is printed indicating that all data files are schema valid. The <code class="highlighter-rouge">--show-pass</code> flag modifies this behavior such that, in addition the the default behavior, a line is printed to stdout for every file that is schema valid indicating it passed the schema adherence check.</p> </blockquote> <p>In looking at the <code class="highlighter-rouge">group_vars/spine.yaml</code> group above. This is because the first dns server in the list which is the value of <code class="highlighter-rouge">dns_servers</code> has a value of the boolean <code class="highlighter-rouge">true</code>.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">bash$ cat schema/schemas/dns.yml</span> <span class="nn">---</span> <span class="s">$schema</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://json-schema.org/draft-07/schema#"</span> <span class="s">$id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">schemas/dns_servers"</span> <span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">DNS</span><span class="nv"> </span><span class="s">Server</span><span class="nv"> </span><span class="s">Configuration</span><span class="nv"> </span><span class="s">schema."</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">dns_servers</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">array"</span> <span class="na">items</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">address</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">format</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ipv4"</span> <span class="na">vrf</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">address"</span> <span class="na">uniqueItems</span><span class="pi">:</span> <span class="no">true</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">dns_servers"</span> </code></pre></div></div> <p>In looking at the schema for dns servers, we see that DNS servers address field must be of type <code class="highlighter-rouge">string</code> and format <code class="highlighter-rouge">ipv4</code> (e.g. IPv4 address). Because the first element in the list of DNS servers has an <code class="highlighter-rouge">address</code> of the boolean <code class="highlighter-rouge">true</code> it is not schema valid.</p> <h3 id="another-example-of-validating-ansible-vars">Another Example of Validating Ansible Vars</h3> <p>Similar to the way that <code class="highlighter-rouge">schema-enforcer validate --show-checks</code> can be used to show which data files will be checked by which schema definitions, the <code class="highlighter-rouge">schema-enforcer ansible --show-checks</code> command can be used to show which Ansible hosts will be checked for adherence to which schema IDs.</p> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-ansible3.png" alt="Schema Enforcer Ansible 3" /></p> <p>From the execution of the command, we can see that 4 hosts were loaded from inventory. This is just what we expect from our earlier examination of the <code class="highlighter-rouge">.ini</code> file which defines Ansible inventory. We just saw how <code class="highlighter-rouge">spine1</code> and <code class="highlighter-rouge">spine2</code> were checked for adherence to both the <code class="highlighter-rouge">schemas/dns_servers</code> and <code class="highlighter-rouge">schemas/interfaces</code> schema definitions, and how the <code class="highlighter-rouge">schema_enforcer_schema_ids</code> var was configured to declare that devices belonging to the <code class="highlighter-rouge">spine</code> group should adhere to those schemas. Lets now examine the <code class="highlighter-rouge">leaf</code> group a little more closely.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">cat ansible/group_vars/leaf.yml</span> <span class="nn">---</span> <span class="na">dns_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.1.1.1"</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.2.2.2"</span> </code></pre></div></div> <p>In the <code class="highlighter-rouge">leaf.yml</code> file, no <code class="highlighter-rouge">schema_enforcer_schema_ids</code> var is configured. There is also no individual data defined at the host level for <code class="highlighter-rouge">leaf1</code> and <code class="highlighter-rouge">leaf2</code> which belong to the <code class="highlighter-rouge">leaf</code> group. This brings up the question, how does <code class="highlighter-rouge">schema-enforcer</code> know to check the leaf switches for adherence to the <code class="highlighter-rouge">schemas/dns_servers</code> schema definition?</p> <p>The default behavior of <code class="highlighter-rouge">schema-enforcer</code> is to map the top level property in a schema definition to vars associated with each Ansible host that have the same name.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">bash$ cat schema/schemas/dns.yml</span> <span class="nn">---</span> <span class="s">$schema</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://json-schema.org/draft-07/schema#"</span> <span class="s">$id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">schemas/dns_servers"</span> <span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">DNS</span><span class="nv"> </span><span class="s">Server</span><span class="nv"> </span><span class="s">Configuration</span><span class="nv"> </span><span class="s">schema."</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">dns_servers</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">array"</span> <span class="na">items</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">object"</span> <span class="na">properties</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">address</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">format</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ipv4"</span> <span class="na">vrf</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">string"</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">address"</span> <span class="na">uniqueItems</span><span class="pi">:</span> <span class="no">true</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">dns_servers"</span> </code></pre></div></div> <p>Because the property defined in the schema definition above is <code class="highlighter-rouge">dns_servers</code>, the matching Ansible host var <code class="highlighter-rouge">dns_servers</code> will be checked for adherence against it.</p> <p>In fact, if we make the following changes to the <code class="highlighter-rouge">leaf</code> group var definition then run <code class="highlighter-rouge">schema-enforcer --show-checks</code>, we can see that devices belonging to the <code class="highlighter-rouge">leaf</code> group are now slated to be checked for adherence to both the <code class="highlighter-rouge">schemas/dns_servers</code> and <code class="highlighter-rouge">schemas/interfaces</code> schema definitions.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">cat ansible/group_vars/leaf.yml</span> <span class="nn">---</span> <span class="na">dns_servers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.1.1.1"</span> <span class="pi">-</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.2.2.2"</span> <span class="na">interfaces</span><span class="pi">:</span> <span class="na">swp01</span><span class="pi">:</span> <span class="na">role</span><span class="pi">:</span> <span class="s">uplink</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/schema-enforcer-launch/schema-enforcer-ansible3.png" alt="Schema Enforcer Ansible 4" /></p> <h2 id="using-schema-enforcer">Using Schema Enforcer</h2> <p>O.K. so you’ve defined schemas for your data, now what? Here are a couple of use cases for Schema Enforcer we’ve found to be “juice worth the squeeze.”</p> <p>1) Use Schema Enforcer in your CI system to validate defined structured data before merging code. Virtually all git version control systems (GitHub, GitLab…etc) allow the ability to configure tests which must pass before code can be merged from a feature branch into the code base. Schema Enforcer can be turned on along side your other tests (unit tests, linters…etc). If your data is not schema valid, the exact reason why the data is not schema valid will be printed to the output of the CI system when the tool is run and the tool will exit with a code of 1 causing the CI system to register a failure. When the CI system sees a failure, it will not allow the merge of data which is not adherent to schema.</p> <p>2) Use it in a pipeline. Say you have YAML structured data which defines the configuration for network devices. You can run schema enforcer as part of a pipeline and run it before automation tooling (Ansible, Python…etc) consumes this data in order to render configurations for devices. If the data isn’t schema valid, the pipeline will fail before rendering configurations and pushing them to devices (or exploding with a stack trace that takes you 30 minutes and lots of googling to troubleshoot).</p> <p>3) Run it after your tooling generates structured data and prints it to a file. In this case, Schema Enforcer can act as a sanity check to ensure that your tooling is dumping correctly structured output.</p> <h2 id="where-are-we-going-next">Where Are We Going Next</h2> <p>We plan to add the support for the following features to Schema Enforcer in the future:</p> <ul> <li>Validation of Nornir inventory attributes</li> <li>Business logic validation</li> </ul> <p>Have a use case for Schema Enforcer? Try it out and let us know what you think! Do you want the ability to write schema definitions in YANG or have another cool idea? We are iterating on the tool and we are open to feedback!</p> <p>-Phillip Simonds</p>Phillip SimondsThese days, most organizations heavily leverage YAML and JSON to store and organize all sorts of data. This is done in order to define variables, be provided as input for generating device configurations, define inventory, and for many other use cases. Both YAML and JSON are very popular because both languages are very flexible and are easy to use. It is relatively easy for users who have little to no experience working with structured data (as well as for very experienced programmers) to use JSON and YAML because the formats do not require users to define a schema in order to define data.Introduction to PromQL2021-02-09T00:00:00+00:002021-02-09T00:00:00+00:00https://blog.networktocode.com/post/promql_for_network_telemetry<p>Time series databases and their query languages are tools with increasing popularity for a Network Automation Engineer. However, sometimes these tools may be overlooked by network operators for more “pressing” day-to-day workflow automation. Time series databases offer valuable network telemetry that will reveal important insights for network operations, such as security breaches, network outages, and slowdowns that degrade the user experience.</p> <p>In this post, we will review the Prometheus Query Language (PromQL) to demonstrate the value and capabilities of processing time series. This review will offer use cases of PromQL for network engineers and data scientists.</p> <h2 id="what-is-prometheus">What is Prometheus?</h2> <p><a href="https://prometheus.io/">Prometheus</a> is an open source systems monitoring and alerting toolkit. As you can see in the figure below, the heart of Prometheus includes a Time Series Database (TSDB) and the PromQL Engine. Exporters run locally on monitored hosts and export local metrics related to device health, such as CPU and memory utilization, and services, such as HTTP. The alert mechanism implemented with Prometheus, triggers alerts based on events and predefined thresholds. Prometheus has a web UI that we will be using in the examples of this post. In addition, the Prometheus measurements can be visualized using <a href="https://grafana.com/">Grafana</a> dashboards.</p> <p><img src="../../../static/images/blog_posts/promql/prometheus.png" alt="" /></p> <p>Source: <a href="https://prometheus.io/docs/introduction/overview/">Prometheus Overview</a></p> <h2 id="what-is-a-tsdb">What is a TSDB?</h2> <p>In simple words, it is a database that stores time series. Then, what is a time series? It is a set of time-stamps and their corresponding data. A TSDB is optimized to store these time series data efficiently, measure changes, and perform calculations over time. PromQL is the language that was built to retrieve data from the Prometheus TSDB. In networking, this could mean tracking the state of an interface or bandwidth utilization over time.</p> <h2 id="why-promql">Why PromQL?</h2> <p>There are several other TSDBs, one of the most well known is <a href="https://www.influxdata.com/">InfluxDB</a>. Both Prometheus TSDB and InfluxDB are excellent tools for telemetry and time series data manipulation. PromQL’s popularity has been growing fast because it is a comprehensive language to consume time series data. Multiple other solutions are starting to support PromQL, such as NewRelic that recently added <a href="https://docs.newrelic.com/docs/query-your-data/explore-query-data/query-builder/use-advanced-promql-style-mode-query-data">support for PromQL</a> and Timescale with <a href="https://blog.timescale.com/blog/promscale-analytical-platform-long-term-store-for-prometheus-combined-sql-promql-postgresql/">Promscale</a>.</p> <p>Now that we have all the prerequisite knowledge we can dive deep into the PromQL data model and dissect language queries.</p> <h2 id="prometheus-data-model">Prometheus Data Model</h2> <p>The first part of the Prometheus data model is the <em>metric name</em>. A metric name is uniquely identified, and it indicates what is being measured. A metric is a dimension of a specific feature. Labels are the second part of the data model. A label is a key-value pair that differentiates sub-dimensions in a metric.</p> <p>Think of a metric, ex. <code class="highlighter-rouge">interface_in_octets</code>, as an object with multiple characteristics, ex., <code class="highlighter-rouge">device_role</code>. As you can see in the figure below, each label can pick a value for this characteristic, i.e. <code class="highlighter-rouge">device_role="leaf"</code>. The combination of metrics and labels return a <em>time series identifier</em>, i.e., a list of tuples that provide the <code class="highlighter-rouge">(timestamp, value)</code> of the object with the specific characteristic. The timestamps are given in Unix time, milliseconds precision and the values that correspond to them are floating point type.</p> <p>As a Network Automation Engineer you can think of many examples of metrics, such as <code class="highlighter-rouge">interface_speed</code>, <code class="highlighter-rouge">bgp_hold_time</code>, <code class="highlighter-rouge">packets_dropped</code>, etc. All these metrics can be characterized by a variety of labels, such as <code class="highlighter-rouge">device_platform</code>, <code class="highlighter-rouge">host</code>, <code class="highlighter-rouge">instance</code>, <code class="highlighter-rouge">interface_name</code> etc.</p> <p><img src="../../../static/images/blog_posts/promql/prometheus-data-model.png" alt="" /></p> <p>With that data model in mind, let us next dissect a query in PromQL.</p> <h2 id="the-anatomy-of-a-query">The anatomy of a query</h2> <p>The simplest form of a PromQL query may include just a metric. This query returns multiple single value vectors, as you can see below. All the applicable labels and value combinations that these labels can be assigned are given as a result of this simple query.</p> <p><img src="../../../static/images/blog_posts/promql/query-metric.png" alt="" /></p> <h3 id="metrics">Metrics</h3> <p>What kind of metrics does PromQL support? There are four kinds of metrics:</p> <ol> <li><strong>Counters</strong>: these are metrics that can <em>only increase</em>, for example: interface counters, API call counters, etc.</li> <li><strong>Gauges</strong>: the values of these metrics can go up and down, for example: bandwidth, latency, packets dropped, etc. Gauges and counters are useful for network engineers because they can measure already existent features of a system.</li> <li><strong>Summaries</strong>: this metric is useful to data scientists and if your application includes data analytics. To use this metric you need have control of what you can measure and drill into additional details. A summary metric aggregates thousands of events to one metric. Specifically it counts observations and sums all the observed values. It can also calculate <a href="https://en.wikipedia.org/wiki/Quantile">quantiles</a> of these values. If you have an application that is being monitored, you can use the summaries for API request durations.</li> <li><strong>Histograms</strong>: this is another metric that is more useful to a data scientist than a network engineer. Histogram metrics can be defined as summaries that are “bucketized”. Specifically they count observations and place them in configurable buckets. A histogram can be used to measure response sizes on an application.</li> </ol> <h3 id="label-filtering">Label Filtering</h3> <p>Now that we know what kinds of metrics we can include in our query, let us review how we can filter the query to retrieve more specific and meaningful results. This can be done with label filtering that includes the following operations:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># equal, returns interface speed for device with name jcy-bb-01</span> interface_speed<span class="o">{</span><span class="nv">device</span><span class="o">=</span><span class="s2">"jcy-bb-01.infra.ntc.com"</span><span class="o">}</span> <span class="c"># not equal, returns the opposite of the above query</span> interface_speed<span class="o">{</span>device!<span class="o">=</span><span class="s2">"jcy-bb-01.infra.ntc.com"</span><span class="o">}</span> <span class="c"># regex-match, matches interface Ethernet{1, 2, 3, 4, 5, 6, 7}</span> interface_speed<span class="o">{</span><span class="nv">interface</span><span class="o">=</span>~<span class="s2">"Ethernet1/[1-7]"</span><span class="o">}</span> <span class="c"># not regex-match, returns the opposite of the above query</span> interface_speed<span class="o">{</span>interface!~<span class="s2">"Ethernet1/[1-7]"</span><span class="o">}</span> </code></pre></div></div> <p>Not only can you use the equal and not equal signs to filter your queries, but you can filter using regular expressions. To learn more about regular expressions for network engineers, check our previous <a href="https://blog.networktocode.com/post/regex_for_network_engineers/">blog</a>.</p> <h2 id="functions">Functions</h2> <p>One of my favorite parts of PromQL are the functions that can manipulate the time series identifiers. Below, I include an example of the function <code class="highlighter-rouge">rate()</code>, that is useful for network metrics, and the function <code class="highlighter-rouge">predict_linear()</code>, that is useful if you perform data analytics.</p> <h3 id="how-fast-does-a-counter-change">How fast does a counter change?</h3> <p>The function <code class="highlighter-rouge">rate()</code> can be used with <em>counter</em> metrics to demonstrate how fast a counter increases. Specifically, it calculates the per second increase for a time period. This is a useful function to the network engineer, since counters are a common metric applied in networks. For example packet counting, interface octets counting are counters and the <code class="highlighter-rouge">rate()</code> function offers useful insights on how these counters increase.</p> <!-- - `irate()`: uses only the last two samples of a time period to calculate how the counter metric has changed. This function reacts faster to changes, for example if you want to alert as fast as possible for an increase in latency, you would prefer `irate` over `rate`. - `increase()`: calculates the increase over a given time period. --> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#per second increase of counter averaged over 5 mins</span> rate<span class="o">(</span>interface_in_octets<span class="o">{</span><span class="nv">device_role</span><span class="o">=</span><span class="s2">"leaf"</span><span class="o">}[</span>5m]<span class="o">)</span> </code></pre></div></div> <!-- # instantaneous rate irate(interface_in_octets{device_role="leaf"}[5m]) # increase over time window increase(interface_in_octets{device_role="leaf"}[5m]) ``` --> <p>The next figure will help you understand the details of how the <code class="highlighter-rouge">rate()</code> function is calculated. The interval $\Delta$t indicates the time interval during which we want to calculate the rate. The <code class="highlighter-rouge">X</code> marks indicate the per second samples that are used to calculate multiple rates per second. The <code class="highlighter-rouge">rate()</code> function averages these calculations during the interval $\Delta$t. If the counter is reset to 0, the <code class="highlighter-rouge">rate()</code> function will extrapolate the sample as can be seen with the blue <code class="highlighter-rouge">X</code> marks.</p> <!-- The `irate()` function is calculating the fraction $\Delta$v / $\Delta$t and finally the `increase()` function returns the increase $\Delta$v. --> <p><img src="../../../static/images/blog_posts/promql/rate.png" alt="" /></p> <h3 id="instance-vs-range-vectors">Instance vs. Range Vectors</h3> <p>You probably have noticed that the example of the <code class="highlighter-rouge">rate()</code> function above, uses a different type of syntax. Specifically it identifies the time series during an interval, from the example above the interval is 5 minutes (<code class="highlighter-rouge">[5m]</code>). This results to a range vector, where the time-series identifier returns the values for a given period, in this case 5 minutes. On the other hand, an instance vector returns one value, specifically the single latest value of a time series. The figures below shows the differences in the results of an instance vector versus a range vector.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#instance vector</span> interface_speed<span class="o">{</span><span class="nv">device</span><span class="o">=</span><span class="s2">"jcy-bb-01.infra.ntc.com"</span><span class="o">}</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/promql/query-instance-vector.png" alt="" /></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#range vector</span> interface_speed<span class="o">{</span><span class="nv">device</span><span class="o">=</span><span class="s2">"jcy-bb-01.infra.ntc.com"</span><span class="o">}[</span>5m] </code></pre></div></div> <p><img src="../../../static/images/blog_posts/promql/query-range-vector.png" alt="" /></p> <p>In the first figure, only one value per vector is returned whereas in the second, multiple values that span in the range of 5 minutes are returned for each vector. The format of these values is: <code class="highlighter-rouge">value@timestamp</code>.</p> <h3 id="offsets">Offsets</h3> <p>You may be wondering: all of this is great, but where is the “time” in my “time-series”? The <code class="highlighter-rouge">offset</code> part of the query can retrieve data for a specific time interval. For example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># interface speed status for the past 24 hrs</span> rate<span class="o">(</span>interface_in_octets<span class="o">{</span><span class="nv">device</span><span class="o">=</span><span class="s2">"jcy-bb-01.infra.ntc.com"</span><span class="o">}[</span>5m] offset 24h<span class="o">)</span> </code></pre></div></div> <p>Here we combine the function <code class="highlighter-rouge">rate()</code>, that samples the <code class="highlighter-rouge">interface_in_octets</code> counter every second for five minutes, with <code class="highlighter-rouge">offset</code> that gives us historical data for the past 24 hours.</p> <!-- ### How fast does a gauge change? Similar functions are available to calculate how fast a gauge metric changes: - `deriv()`: calculates the per second derivative of a set of series under a time window. For example, if you would like to check how fast the disc usage in bytes changes, either increases or decreases, you would use the derivative. - `delta()`: calculates the difference between first and last value over time window. If you want to see actual change in values, increase or decrease, you can use delta. ```bash #deriv deriv(bgp_total_messages{device_role="router"}[15m]) ``` ![](../../../static/images/blog_posts/promql/deriv.png) ```bash #delta delta(bgp_total_messages{device_role="router"}[15m]) ``` ![](../../../static/images/blog_posts/promql/delta.png) In the figures above, you can observe how smoothly the derivative changes versus the abrupt changes of the delta function. --> <h3 id="can-i-predict-the-next-24-hours">Can I predict the next 24 hours?</h3> <p>Of course! PromQL provides the function <code class="highlighter-rouge">predict_linear()</code>, a simple machine learning model that predicts the value of a gauge in a given amount of time in the future, by using linear regression. This function is of more interest to a data scientist that wants to create forecasting models. For example, if you want to predict the disk usage in bytes within the next hour based on historic data, you would use the following query:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#predict disk usage bytes in an hour, using the last 15 mins of data</span> predict_linear<span class="o">(</span>demo_disk_usage_bytes<span class="o">{</span><span class="nv">job</span><span class="o">=</span><span class="s2">"demo"</span><span class="o">}[</span>15m], 3600<span class="o">)</span> </code></pre></div></div> <p>Linear regression fits a linear function to a set of random data points. This is achieved by searching for all possible values for the variables a, b that define a linear function <code class="highlighter-rouge">f(x)=ax+b</code>. The line that minimizes the mean Euclidean distance of all these data points is the result of the linear regression model, as you can see in the image below:</p> <p><img src="../../../static/images/blog_posts/promql/linear-regression.png" alt="" /></p> <h2 id="aggregation">Aggregation</h2> <p>PromQL queries can be highly dimensional. This means that one query can return a set of time series identifiers for all the combinations of labels, as you can see below:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#multi-dimensional query</span> rate<span class="o">(</span>demo_api_request_duration_seconds_count<span class="o">{</span><span class="nv">job</span><span class="o">=</span><span class="s2">"demo"</span><span class="o">}[</span>5m]<span class="o">)</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/promql/multi-dimensional.png" alt="" /></p> <p>What if you want to reduce the dimensions to a more meaningful result, for example the sum of all the API request durations in seconds? This would result in a single-dimension query that is the result of adding multiple instance vectors together:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#one-dimensional query, add instance vectors</span> <span class="nb">sum</span><span class="o">(</span>rate<span class="o">(</span>demo_api_request_duration_seconds_count<span class="o">{</span><span class="nv">job</span><span class="o">=</span><span class="s2">"demo"</span><span class="o">}[</span>5m]<span class="o">))</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/promql/one-dimensional.png" alt="" /></p> <p>You may choose to aggregate over specific dimensions using labels and the function <code class="highlighter-rouge">by()</code>. In the example below, we perform a sum over all instances, paths, and jobs. Note the reduction of the number of vectors returned:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># multi-dimensional query - by()</span> <span class="nb">sum </span>by<span class="o">(</span>instance, path, job<span class="o">)</span> <span class="o">(</span>rate<span class="o">(</span>demo_api_request_duration_seconds_count<span class="o">{</span><span class="nv">job</span><span class="o">=</span><span class="s2">"demo"</span><span class="o">}[</span>5m]<span class="o">))</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/promql/sum-by.png" alt="" /></p> <p>We can perform the same query excluding labels using the function <code class="highlighter-rouge">without()</code>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sum </span>without<span class="o">(</span>method, status<span class="o">)</span> <span class="o">(</span>rate<span class="o">(</span>demo_api_request_duration_seconds_count<span class="o">{</span><span class="nv">job</span><span class="o">=</span><span class="s2">"demo"</span><span class="o">}[</span>5m]<span class="o">))</span> </code></pre></div></div> <p>This results to the same set of instance vectors:</p> <p><img src="../../../static/images/blog_posts/promql/sum-without.png" alt="" /></p> <p>Additional aggregation over dimensions can be done with the following functions:</p> <ul> <li><code class="highlighter-rouge">min()</code>: selects the minimum of all values within an aggregated group.</li> <li><code class="highlighter-rouge">max()</code>: selects the maximum of all values within an aggregated group.</li> <li><code class="highlighter-rouge">avg()</code>: calculates the average (arithmetic mean) of all values within an aggregated group.</li> <li><code class="highlighter-rouge">stddev()</code>: calculates the standard deviation of all values within an aggregated group.</li> <li><code class="highlighter-rouge">stdvar()</code>: calculates the standard variance of all values within an aggregated group.</li> <li><code class="highlighter-rouge">count()</code>: calculates the total number of series within an aggregated group.</li> <li><code class="highlighter-rouge">count_values()</code>: calculates number of elements with the same sample value.</li> </ul> <h2 id="conclusion">Conclusion</h2> <p>Thank you for taking this journey with me, learning about the time series query language, PromQL. There are many more features to this language such as arithmetic, sorting, set functions etc. I hope that this post has given you the opportunity to understand the basics of PromQL, see the value of telemetry and TSDBs, and that it has increased your curiosity to learn more.</p> <p>-Xenia</p> <h2 id="useful-resources">Useful Resources</h2> <ul> <li><a href="https://demo.promlabs.com/new">PromQL Demo</a>, official demo where you can practice PromQL queries.</li> <li><a href="http://blog.networktocode.com/post/prometheus_alerting/">Alerting with Prometheus</a> blog by Josh VanDeraa, Network To Code.</li> <li><a href="http://blog.networktocode.com/post/monitoring_websites_with_telegraf_and_prometheus/">Monitoring Websites with Telegraf and Prometheus</a> blog by Josh VanDeraa, Network To Code.</li> </ul>Xenia MountrouidouTime series databases and their query languages are tools with increasing popularity for a Network Automation Engineer. However, sometimes these tools may be overlooked by network operators for more “pressing” day-to-day workflow automation. Time series databases offer valuable network telemetry that will reveal important insights for network operations, such as security breaches, network outages, and slowdowns that degrade the user experience.Regex for Network Engineers2021-01-26T00:00:00+00:002021-01-26T00:00:00+00:00https://blog.networktocode.com/post/regex_for_network_engineers<p>Every IT professional will encounter Regex at some point in their career. Contrary to how it looks, Regex is not just some funny strings of random characters. You can in fact find it powering some of the world’s most critical technology infrastructure. Regex is supported out of the box in many different technologies, so learning just the basics can really accelerate your automation workflow. This post will teach you the basics and give you some real-world applications of Regex for Network Engineers.</p> <h2 id="sounds-good-but-what-is-regex">Sounds good! But what is Regex?</h2> <p>Well, this is Regex:</p> <p><code class="highlighter-rouge">[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}</code></p> <p>Yeah, I know, looks kind of like hieroglyphics, right? Well, this may look like a foreign language, especially if you are just starting out, but as with any kind of new language, we need to work through the basics first.</p> <h2 id="okay-for-real-this-time-what-is-regex">Okay, for real this time. What is Regex?</h2> <p>Regex, or regular expression, is a pattern matching engine used to find or parse text and outputs for specified patterns. Regex is built into tools like Vim, grep, and even Python! We can use Regex to parse text and outputs so we can get only the data we need.</p> <h2 id="why-is-it-important-for-network-engineers">Why is it important for Network Engineers?</h2> <p>Well, there are patterns all around us in the networking world. A MAC address is nothing more than a pattern of some characters that can be <code class="highlighter-rouge">A-F</code> and <code class="highlighter-rouge">0-9</code> followed by a period or colon (depending on who you ask). And take the IPv4 address for example. If you had to teach someone how to spot an IPv4 address in a random list of characters, what would you tell them?</p> <p>Well, you might say an IPv4 address is:</p> <ul> <li>One to three digits followed by a period</li> <li>One to three digits followed by another period</li> <li>One to three digits followed by <em>another</em> period</li> <li>One to three digits <strong>NOT</strong> followed by a period.</li> </ul> <p>Is this starting to make sense?</p> <p><code class="highlighter-rouge">[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}</code></p> <p>Well, hold your horses, we’re almost there. Let’s quickly go over some basics.</p> <h2 id="basics">Basics</h2> <p>Many people, myself included, use the website <a href="https://regex101.com/">Regex101</a> when making and testing new patterns. With this we can actually see what the Regex engine is thinking when it matches our patterns.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex1.png" alt="" /></p> <p>You can see that since I typed “text” in the <code class="highlighter-rouge">Regular Expression</code> box, it will highlight only the words that match “text”. Regex101 also includes a handy <code class="highlighter-rouge">Explanation</code> and other information on the right.</p> <p>Let’s go into the different syntax you’ll be using to create Regex patterns.</p> <h3 id="character-classes--">Character classes [ ]</h3> <p>First things first, we need to learn about character classes. A character class in Regex is represented by brackets <code class="highlighter-rouge">[]</code>, and it is where you put all the characters like <code class="highlighter-rouge">0-9</code> or <code class="highlighter-rouge">A-Z</code> that you want to match.</p> <h3 id="quantifiers--">Quantifiers { }</h3> <p>A quantifier in Regex is represented by curly braces <code class="highlighter-rouge">{}</code>, which are used to express how many times you want to match your characters. For example, if you want something to match one to three times, you use <code class="highlighter-rouge">{1,3}</code>. The comma <code class="highlighter-rouge">,</code> acts as a “through” in Regex.</p> <h3 id="escape-characters-">Escape characters \</h3> <p>As you may have noticed, Regex uses a lot of regular everyday characters like <code class="highlighter-rouge">.[]{}/</code> as their pattern syntax. But what if you want to match <em>literally</em> something like the period <code class="highlighter-rouge">.</code>? Well, any characters like that can be matched by adding an escape character before the character. In Regex this is represented by the backslash <code class="highlighter-rouge">\</code>.</p> <p>So <code class="highlighter-rouge">.</code> becomes <code class="highlighter-rouge">\.</code> to match. And, yes, you can even escape character the escape character by doing a double backslash <code class="highlighter-rouge">\\</code>.</p> <h2 id="put-these-skills-to-work">Put these skills to work!</h2> <p>So that is a <em>very</em> basic overview of Regex. So enough talking—let’s get to work! Let’s look at our IPv4 instructions again:</p> <ul> <li>One to three digits followed by a period</li> <li>One to three digits followed by another period</li> <li>One to three digits followed by <em>another</em> period</li> <li>One to three digits <strong>NOT</strong> followed by a period.</li> </ul> <p>For <code class="highlighter-rouge">digits</code> we’re going to make a capture group for all possible digits. This would be written as <code class="highlighter-rouge">[0-9]</code>.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex2.png" alt="" /></p> <p>For <code class="highlighter-rouge">one to three</code> we’re going to use our handy quantifier to express that range. This would be written as <code class="highlighter-rouge">{1,3}</code>.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex3.png" alt="" /></p> <p>Finally, the period <code class="highlighter-rouge">.</code> is a special Regex syntax, so we’ll need to use the escape character <code class="highlighter-rouge">\</code>. This would be written as <code class="highlighter-rouge">\.</code>.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex4.png" alt="" /></p> <p>Okay, now let’s just repeat that—don’t forget to omit the last period <code class="highlighter-rouge">.</code>!</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex5.png" alt="" /></p> <p>Look at that!</p> <p><code class="highlighter-rouge">[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}</code></p> <p>Not so scary now, is it? So let’s put this new pattern to work. The following is a simple <code class="highlighter-rouge">show ip route</code> that, of course, contains IPv4 addresses.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex6.png" alt="" /></p> <p>Let’s drop our pattern and see what happens.</p> <p><img src="../../../static/images/blog_posts/regex_neteng/regex7.png" alt="" /></p> <p>SUCCESS!! Regex with our pattern can now match any IPv4 address we throw at it!</p> <h2 id="conclusion">Conclusion</h2> <p>Thanks for reading and spending some time learning Regex with me. This, of course, is a <em>very</em> basic overview that just scratches the surface. We can continue to improve our pattern to make it more and more accurate at matching IPv4 addresses. For example, <code class="highlighter-rouge">999.999.999.999</code> would be a valid match with our pattern, but not a valid IPv4 address. This article is just meant to be a good jumping off point for you to make your own patterns.</p> <p>And I hope you do feel empowered to go out and make your own Regex patterns! There are so many novel and unique ways Network Engineers are using Regex today. One of the most popular use case is parsing the CLI outputs from our network devices. There are many open sourced projects that need custom Regex patterns for the many different network devices out in the wild. We’ll go through some of those and how you can contribute to projects like <a href="https://github.com/networktocode/ntc-templates">NTC Templates</a> and <a href="https://developer.cisco.com/docs/genie-docs/">Cisco’s Genie Parsers</a> in a later post.</p> <p>In the meantime, can you work out a pattern to match MAC addresses? Try it out, and let me know your solution in the comments below!</p>Robert SchneiderEvery IT professional will encounter Regex at some point in their career. Contrary to how it looks, Regex is not just some funny strings of random characters. You can in fact find it powering some of the world’s most critical technology infrastructure. Regex is supported out of the box in many different technologies, so learning just the basics can really accelerate your automation workflow. This post will teach you the basics and give you some real-world applications of Regex for Network Engineers.Jinja2: Assemble Strategy2021-01-12T00:00:00+00:002021-01-12T00:00:00+00:00https://blog.networktocode.com/post/Jinja2_assemble_strategy<p>Because you are reading this post, you’re likely aware of the great power of the Jinja2 templating language. You might have turned an entire data center configuration in one huge complex template, doing crazy conditionals to render some lines and skip some others. Spines or leaves devices do not make any difference to you, as long as you have a <code class="highlighter-rouge">device_role</code> variable which helps you to build the right config. Multi-vendor is not a thing for you anymore - Juniper, Cisco, Nexus (…you name them) all together in your <code class="highlighter-rouge">super_master.j2</code>. One template to rule them all! Hundreds and hundreds of lines that do the magic and make you scream every time <code class="highlighter-rouge">template error while templating string</code> or <code class="highlighter-rouge">UndefinedVariable</code> pops out! You end up digging into the code for hours, commenting blocks of lines, and trying to run it again..and again. Rolling back git commits, making the effort to remember when the last time was that the template rendered properly. What about when you needed to add a few config lines just for a specific bunch of devices or you tried to port your template and reuse it for another project?</p> <p>You suddenly realize that you need a different design to make your template more scalable so you will not go crazy every time you need to work on it. Let’s explore what options we have and what the best approach might be.</p> <h2 id="single-template">Single template</h2> <p>Let’s use the previous case as an example. We will leverage the Ansible rendering engine. Remember, this is a templating strategy, so the below examples are also true for all those applications which rely on Jinja as template language, such as SALT, python, etc. Based on two groups of devices - <code class="highlighter-rouge">spines</code> and <code class="highlighter-rouge">leaves</code> you could build a single <code class="highlighter-rouge">base.j2</code> such as:</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">for</span> <span class="nv">iface</span> <span class="ow">in</span> <span class="nv">interfaces</span> <span class="cp">%}</span> interface <span class="cp">{{</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'Management'</span> <span class="ow">in</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">%}</span> vrf forwarding mgmt <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'loopback'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">%}</span> mtu <span class="cp">{{</span> <span class="nv">iface</span><span class="p">[</span><span class="s1">'mtu'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> [...] ! <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'leaves'</span> <span class="ow">in</span> <span class="nv">group_names</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vxlan</span> <span class="ow">in</span> <span class="nv">vxlans</span> <span class="cp">%}</span> interface <span class="cp">{{</span> <span class="nv">vxlan</span><span class="p">[</span><span class="s1">'interface'</span><span class="p">]</span> <span class="cp">}}</span> vxlan source-interface <span class="cp">{{</span> <span class="nv">vxlan</span><span class="p">[</span><span class="s1">'source_update'</span><span class="p">]</span> <span class="cp">}}</span> vxlan udp-port <span class="cp">{{</span> <span class="nv">vxlan</span><span class="p">[</span><span class="s1">'port'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vlan_vni</span> <span class="ow">in</span> <span class="nv">vxlan</span><span class="p">[</span><span class="s1">'vlan_vni'</span><span class="p">]</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vlan</span><span class="p">,</span> <span class="nv">vni</span> <span class="ow">in</span> <span class="nv">vlan_vni.items</span><span class="p">()</span> <span class="cp">%}</span> vxlan vlan <span class="cp">{{</span> <span class="nv">vlan</span> <span class="cp">}}</span> vni <span class="cp">{{</span> <span class="nv">vni</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> [...] <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> ! <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'spines'</span> <span class="ow">in</span> <span class="nv">group_names</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">peer</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'peers'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> peer-group neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> remote-as <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'remote_as'</span><span class="p">]</span> <span class="cp">}}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> maximum-routes <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'max_routes'</span><span class="p">]</span> <span class="cp">}}</span> warning-only <span class="cp">{%</span> <span class="k">if</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'update_source'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> update-source <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'update_source'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'ebgp_multihop'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> ebgp-multihop <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'ebgp_multihop'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'next_hop_unchanged'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> next-hop-unchanged <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'attributes'</span><span class="p">][</span><span class="s1">'next_hop_self'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">peer</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> next-hop-self <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> [...] <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> </code></pre></div></div> <p>Well…that was a lot of code to read just for an interface, VXLAN, and BGP configuration. Now try to imagine building a full running-config how complex it would become. Catching an error becomes tricky. Making future implementations feels like trying to move through a minefield. Let’s see how we can improve our template design.</p> <h2 id="include-strategy">Include strategy</h2> <p>Jinja comes to the rescue with <a href="https://jinja.palletsprojects.com/en/2.11.x/templates/#include">include</a> tag.</p> <p>We can break up our <code class="highlighter-rouge">base.j2</code> in small chunks of templates and move them under the appropriate folder:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── base.j2 ├── bgp │ └── bgp.j2 ├── interfaces │ └── interfaces.j2 └── vxlan └── vxlan.j2 </code></pre></div></div> <p>so, our <code class="highlighter-rouge">base.j2</code> will look like this:</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">include</span> <span class="err">`</span><span class="nv">interface</span><span class="o">/</span><span class="nv">interface.j2</span><span class="err">`</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'leaves'</span> <span class="ow">in</span> <span class="nv">group_names</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">include</span> <span class="err">`</span><span class="nv">vxlan</span><span class="o">/</span><span class="nv">vxlan.j2</span><span class="err">`</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="s1">'spines'</span> <span class="ow">in</span> <span class="nv">group_names</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">include</span> <span class="err">`</span><span class="nv">bgp</span><span class="o">/</span><span class="nv">bgp.j2</span><span class="err">`</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> </code></pre></div></div> <p>Much better, isn’t it? Every template is now properly organized under its own folder and file (i.e. interface under <code class="highlighter-rouge">interface/interface.j2</code>) and we included them into base template. By still applying the group conditional logic, we can have the desired config for each device. As you can see, the code becomes more readable, easier to implement (if we need to update a BGP template, we will work only on <code class="highlighter-rouge">bgp.j2</code>), easier to troubleshoot, and more portable.</p> <p>Great! But what if this can be further simplified? Get ready, minimalists!</p> <h2 id="jinja-assemble-strategy">Jinja assemble strategy</h2> <p>Still using the <code class="highlighter-rouge">include</code> tag, we can assemble our configuration, plugging in or out parts of templates and looping through a list of includes for each group of device.</p> <p>Assuming we are still using Ansible, we can group devices per role and have a <code class="highlighter-rouge">group_vars</code> folder that looks like this:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── leaves │ └── main.yml └── spines └── main.yml </code></pre></div></div> <p>Where <code class="highlighter-rouge">leaves/main.yml</code>…</p> <div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">assemble</span><span class="pi">:</span> <span class="pi">-</span> <span class="s1">'</span><span class="s">interface/interface.j2'</span> <span class="pi">-</span> <span class="s1">'</span><span class="s">vxlan/vxlan.j2'</span> </code></pre></div></div> <p>…and <code class="highlighter-rouge">spines/main.yml</code></p> <div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">assemble</span><span class="pi">:</span> <span class="pi">-</span> <span class="s1">'</span><span class="s">interface/interface.j2'</span> <span class="pi">-</span> <span class="s1">'</span><span class="s">bgp/bgp.j2'</span> </code></pre></div></div> <p>Defining what template we want to include in our assemble process, and taking advantage of <code class="highlighter-rouge">group_vars</code>, our <code class="highlighter-rouge">base.j2</code> will contain just a simple for loop:</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">for</span> <span class="nv">item</span> <span class="ow">in</span> <span class="nv">assemble</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">include</span> <span class="nv">item</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> </code></pre></div></div> <p>We can now plug in or out templates by just appending or removing a new include under our lists. Our <code class="highlighter-rouge">base.j2</code> will loop through the lists based on <code class="highlighter-rouge">group_vars</code> and assemble the final config.</p> <p>That’s all. It could not be simpler than that!</p> <p>By the way, if you ever had troubles with whitespaces or indentation in Jinja (…and I am sure you have!) make sure to check <a href="../../post/whitespace-control-in-jinja-templates">this</a> out!</p> <p>Federico</p>Federico OlivieriBecause you are reading this post, you’re likely aware of the great power of the Jinja2 templating language. You might have turned an entire data center configuration in one huge complex template, doing crazy conditionals to render some lines and skip some others. Spines or leaves devices do not make any difference to you, as long as you have a device_role variable which helps you to build the right config. Multi-vendor is not a thing for you anymore - Juniper, Cisco, Nexus (…you name them) all together in your super_master.j2. One template to rule them all! Hundreds and hundreds of lines that do the magic and make you scream every time template error while templating string or UndefinedVariable pops out! You end up digging into the code for hours, commenting blocks of lines, and trying to run it again..and again. Rolling back git commits, making the effort to remember when the last time was that the template rendered properly. What about when you needed to add a few config lines just for a specific bunch of devices or you tried to port your template and reuse it for another project?Using NetBox for Ansible Source of Truth2021-01-05T00:00:00+00:002021-01-05T00:00:00+00:00https://blog.networktocode.com/post/netbox_as_ansible_sot<p>This content was originally posted on the <a href="https://www.ansible.com/blog/using-netbox-for-ansible-source-of-truth">Ansible Blog</a> on 2020-12-08. Here you will learn about NetBox at a high level, how it works to become a Source of Truth (SoT), and look into the use of the Ansible Content Collection, which is available on Ansible Galaxy. The goal is to show some of the capabilities that make NetBox a terrific tool and will be using NetBox as your network Source of Truth for automation!</p> <h2 id="source-of-truth">Source of Truth</h2> <p>Why a Source of Truth? The Source of Truth is where you go to get the intended state of the device. There does not need to be a single Source of Truth, but you should have a single Source of Truth per data domain, often referred to as the System of Record (SoR). For example, if you have a database that maintains your physical sites that is used by teams outside of the IT domain, that should be the Source of Truth on physical sites. You can aggregate the data from the physical site Source of Truth into other data sources for automation. Just be aware that when it comes time to collect data, then it should come from that other tool.</p> <p>The first step in creating a network automation framework is to identify the Source of Truth for the data, which will be used in future automations. Oftentimes for a traditional network, the device itself has been considered the SoT. Reading the configuration off of the device each time you need a configuration data point for automation is inefficient, and presumes that the device configuration is as intended, not simply left there in troubleshooting or otherwise inadvertently left. When it comes to providing data to teams outside of the network organization, exposing an API can help to speed up gathering data without having to check in with the device first.</p> <h2 id="netbox">NetBox</h2> <p>For a Source of Truth, one popular open source choice is NetBox. From the primary documentation site https://netbox.readthedocs.io, “NetBox is an open source web application designed to help manage and document computer networks”. NetBox is currently designed to help manage your:</p> <ul> <li>DCIM (Data Center Infrastructure Management)</li> <li>IPAM (IP Address Management)</li> <li>Data Circuits</li> <li>Connections (Network, console, and power)</li> <li>Equipment racks</li> <li>Virtualization</li> <li>Secrets</li> </ul> <p>Since NetBox is an IPAM tool, there are misconceptions at times about what NetBox is able to do. To be clear, NetBox is not:</p> <ul> <li>Network monitoring</li> <li>DNS server</li> <li>RADIUS server</li> <li>Configuration management</li> <li>Facilities management</li> </ul> <h2 id="why-netbox">Why NetBox?</h2> <p>NetBox is a tool that is built on many common Python based open source tools, using Postgres for the backend database and Python Django for the back-end API and front-end UI. The API is extremely friendly as it supports CRUD (Create, Read, Update, Delete) operations and is fully documented with Swagger documentation. The NetBox Collection helps with several aspects of NetBox including an inventory plugin, lookup plugin, and several modules for updating data in NetBox.</p> <p>NetBox gives a modern UI from the point of view of a network organization to help document IP addressing, while keeping the primary emphasis on network devices, system infrastructure, and virtual machines. This makes it ideal to use as your Source of Truth for automating.</p> <p>NetBox itself does not do any scanning of network resources. It is intended to have humans maintain the data as this is going to be the Source of Truth. It represents what the environment should look like.</p> <h2 id="ansible-content-collection-for-netbox">Ansible Content Collection for NetBox</h2> <p>You will find the Collection within the netbox-community <a href="https://github.com/netbox-community/">GitHub organization</a>. Here you find a <a href="https://github.com/netbox-community/netbox-docker">Docker container image</a>, <a href="https://github.com/netbox-community/devicetype-library">device-type library</a>, <a href="https://github.com/netbox-community/reports">community generated NetBox reports</a>, and <a href="https://github.com/netbox-community/netbox">source code for NetBox</a> itself.</p> <p>If you are unfamiliar with what an Ansible Content Collection is, please watch this brief <a href="https://youtu.be/WOcqhk7TdYc">YouTube video</a>.</p> <p>The Galaxy link for the Collection is at <a href="https://galaxy.ansible.com/netbox/netbox">https://galaxy.ansible.com/netbox/netbox</a>.</p> <p>The NetBox Collection allows you to get started quickly in adding information into a NetBox instance. The only requirements are to supply an API key and a URL to get started. With this Collection, a base inventory, and a NetBox environment you are able to get a Source of Truth populated very quickly.</p> <p>Let’s walk through the base setup to get to a place where you are starting to use the NetBox Inventory Plugin as your Ansible inventory. First is the example group_vars/all.yml file that will have the list of items to be used with the tasks.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">site_list</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“NYC”</span> <span class="na">time_zone</span><span class="pi">:</span> <span class="s">America/New_York</span> <span class="na">status</span><span class="pi">:</span> <span class="s">Active</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“CHI”</span> <span class="na">time_zone</span><span class="pi">:</span> <span class="s">America/Chicago</span> <span class="na">status</span><span class="pi">:</span> <span class="s">Active</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“RTP”</span> <span class="na">time_zone</span><span class="pi">:</span> <span class="s">America/New_York</span> <span class="na">status</span><span class="pi">:</span> <span class="s">Active</span> <span class="na">manufacturers</span><span class="pi">:</span> <span class="c1"># In alphabetical order</span> <span class="pi">-</span> <span class="s">Arista</span> <span class="pi">-</span> <span class="s">Cisco</span> <span class="pi">-</span> <span class="s">Juniper</span> <span class="na">device_types</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">model</span><span class="pi">:</span> <span class="s">“ASAv”</span> <span class="na">manufacturer</span><span class="pi">:</span> <span class="s">“Cisco”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“asav”</span> <span class="na">part_number</span><span class="pi">:</span> <span class="s">“asav”</span> <span class="na">Full_depth</span><span class="pi">:</span> <span class="s">False</span> <span class="pi">-</span> <span class="na">model</span><span class="pi">:</span> <span class="s">“CSR1000v”</span> <span class="na">manufacturer</span><span class="pi">:</span> <span class="s">“Cisco”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“csr1000v”</span> <span class="na">part_number</span><span class="pi">:</span> <span class="s">“csr1000v”</span> <span class="na">Full_depth</span><span class="pi">:</span> <span class="s">False</span> <span class="pi">-</span> <span class="na">model</span><span class="pi">:</span> <span class="s">“vEOS”</span> <span class="na">manufacturer</span><span class="pi">:</span> <span class="s">“Arista”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“veos”</span> <span class="na">part_number</span><span class="pi">:</span> <span class="s">“veos”</span> <span class="na">Full_depth</span><span class="pi">:</span> <span class="s">False</span> <span class="pi">-</span> <span class="na">model</span><span class="pi">:</span> <span class="s">“vSRX”</span> <span class="na">manufacturer</span><span class="pi">:</span> <span class="s">“Juniper”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“vsrx”</span> <span class="na">part_number</span><span class="pi">:</span> <span class="s">“vsrx”</span> <span class="na">Full_depth</span><span class="pi">:</span> <span class="s">False</span> <span class="na">platforms</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“ASA”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“asa”</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“EOS”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“eos”</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“IOS”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“ios”</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">“JUNOS”</span> <span class="na">slug</span><span class="pi">:</span> <span class="s">“junos”</span> </code></pre></div></div> <p>The first step is to create a site. Since NetBox models physical gear, you install equipment at a physical location. Whether that is in your own facilities or inside of a cloud, this is a site. The module for this is the netbox.netbox.netbox_site module. A task in the playbook may be:</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">10:</span><span class="nv"> </span><span class="s">SETUP</span><span class="nv"> </span><span class="s">SITES"</span> <span class="s">netbox.netbox.netbox_site</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s">“{{ site_list }}</span></code></pre></figure> <p>The next two pieces are the base to add devices to NetBox. In order to create a specific device, you also need to have the device type and manufacturer in your NetBox instance. To do this there are specific modules available to create them. Platforms will help to identify what OS the device is. I recommend that you use what your automation platform is using—something like IOS, NXOS, and EOS are good choices and should match up to your ansible_network_os choices. These tasks look like the following:</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">20:</span><span class="nv"> </span><span class="s">SETUP</span><span class="nv"> </span><span class="s">MANUFACTURERS"</span> <span class="s">netbox.netbox.netbox_manufacturer</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">manufacturer</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">manufacturers</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">loop_var</span><span class="pi">:</span> <span class="s">manufacturer</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">30:</span><span class="nv"> </span><span class="s">SETUP</span><span class="nv"> </span><span class="s">DEVICE</span><span class="nv"> </span><span class="s">TYPES"</span> <span class="s">netbox.netbox.netbox_device_type</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">model</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type.model</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">manufacturer</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type.manufacturer</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">slug</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type.slug</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">part_number</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type.part_number</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">is_full_depth</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type.full_depth</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_types</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">loop_var</span><span class="pi">:</span> <span class="s">device_type</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_type['model']</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">40:</span><span class="nv"> </span><span class="s">SETUP</span><span class="nv"> </span><span class="s">PLATFORMS"</span> <span class="s">netbox.netbox.netbox_platform</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">platform.name</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">slug</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">platform.slug</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">platforms</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">loop_var</span><span class="pi">:</span> <span class="s">platform</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">platform['name']</span><span class="nv"> </span><span class="s">}}"</span></code></pre></figure> <p>At this stage you are set to add devices and device information to NetBox. The following tasks leverage the ansible_facts that Ansible automatically gathers. So for these particular device types, no additional parsing/data gathering is required outside of using Ansible to gather facts. In this example for adding a device, you will notice custom_fields. A nice extension of NetBox is that if there is not a field already defined, you can set your own fields and use them within the tool.</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">100:</span><span class="nv"> </span><span class="s">NETBOX</span><span class="nv"> </span><span class="s">&gt;&gt;</span><span class="nv"> </span><span class="s">ADD</span><span class="nv"> </span><span class="s">DEVICE</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">NETBOX"</span> <span class="s">netbox.netbox.netbox_device</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">device_type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_model']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">platform</span><span class="pi">:</span> <span class="s">IOS</span> <span class="c1"># May be able to use a filter to define in future</span> <span class="na">serial</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_serialnum']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">status</span><span class="pi">:</span> <span class="s">Active</span> <span class="na">device_role</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">get_role_from_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">site</span><span class="pi">:</span> <span class="s">“ANSIBLE_DEMO_SITE"</span> <span class="na">custom_fields</span><span class="pi">:</span> <span class="na">code_version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_version']</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">110:</span><span class="nv"> </span><span class="s">NETBOX</span><span class="nv"> </span><span class="s">&gt;&gt;</span><span class="nv"> </span><span class="s">ADD</span><span class="nv"> </span><span class="s">INTERFACES</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">NETBOX"</span> <span class="s">netbox.netbox.netbox_device_interface</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">device</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item.key</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">form_factor</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item.key</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">get_interface_type</span><span class="nv"> </span><span class="s">}}"</span> <span class="c1"># Define types</span> <span class="na">mac_address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item.value.macaddress</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">ansible.netcommon.hwaddr</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">state</span><span class="pi">:</span> <span class="s">present</span> <span class="na">with_dict</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_interfaces']</span><span class="nv"> </span><span class="s">}}"</span></code></pre></figure> <p>Once you have the interfaces you can add in IP address information that is included in the ansible_facts data, I show three steps. First is to add a temporary interface (TASK 200), then add the IP address (TASK 210), and finally associate the IP address to the device (TASK 220).</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">200:</span><span class="nv"> </span><span class="s">NETBOX</span><span class="nv"> </span><span class="s">&gt;&gt;</span><span class="nv"> </span><span class="s">Add</span><span class="nv"> </span><span class="s">temporary</span><span class="nv"> </span><span class="s">interface"</span> <span class="s">netbox.netbox.netbox_device_interface</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">device</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Temporary_Interface</span> <span class="na">form_factor</span><span class="pi">:</span> <span class="s">Virtual</span> <span class="na">state</span><span class="pi">:</span> <span class="s">present</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">210:</span><span class="nv"> </span><span class="s">NETBOX</span><span class="nv"> </span><span class="s">&gt;&gt;</span><span class="nv"> </span><span class="s">ADD</span><span class="nv"> </span><span class="s">IP</span><span class="nv"> </span><span class="s">ADDRESS</span><span class="nv"> </span><span class="s">OF</span><span class="nv"> </span><span class="s">ANSIBLE</span><span class="nv"> </span><span class="s">HOST"</span> <span class="s">netbox.netbox.netbox_ip_address</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">family</span><span class="pi">:</span> <span class="m">4</span> <span class="na">address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_host</span><span class="nv"> </span><span class="s">}}/24"</span> <span class="na">status</span><span class="pi">:</span> <span class="s">active</span> <span class="na">interface</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Temporary_Interface</span> <span class="na">device</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">220:</span><span class="nv"> </span><span class="s">NETBOX</span><span class="nv"> </span><span class="s">&gt;&gt;</span><span class="nv"> </span><span class="s">ASSOCIATE</span><span class="nv"> </span><span class="s">IP</span><span class="nv"> </span><span class="s">ADDRESS</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">DEVICE"</span> <span class="s">netbox.netbox.netbox_device</span><span class="pi">:</span> <span class="na">netbox_url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_URL')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">netbox_token</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('ENV',</span><span class="nv"> </span><span class="s">'NETBOX_API_KEY')</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">data</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">device_type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_model']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">platform</span><span class="pi">:</span> <span class="s">IOS</span> <span class="na">serial</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['net_serialnum']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">status</span><span class="pi">:</span> <span class="s">Active</span> <span class="na">primary_ip4</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_host</span><span class="nv"> </span><span class="s">}}/24"</span></code></pre></figure> <h2 id="ansible-inventory-source">Ansible Inventory Source</h2> <p>At this point you have NetBox populated with all of your devices that were in your static inventory. It is now time to make the move to using NetBox as the Source of Truth for your Ansible dynamic inventory plugin. This way you don’t have to keep finding all of the projects that need to get updated when you make a change to the environment. You just need to change your Source of Truth database - NetBox.</p> <p>You define which inventory plugin to use with a YAML file that defines the characteristics of how to configure your intended use of the plugin. Below is an example, showing you are able to query many components of NetBox for use within your Ansible inventory. You may wish to only make an update to your access switches? Use the query_filters key to define what NetBox API searches should be executed. Take a look at the plugin documentation for updated supported parameters on <a href="https://github.com/netbox-community/ansible_modules">GitHub</a> or <a href="https://netbox-ansible-collection.readthedocs.io/en/latest/">ReadTheDocs</a>. The compose key allows you to pass in additional variables to be used by Ansible, as such the platform from above would be used with the ansible_network_os key. This is where you see the definition and what would get passed from the inventory source.</p> <p>This definition also has groups created based on the device_roles that are defined in NetBox and the platforms. So you would be able to access all platforms_ios devices or platforms_eos as an example, based on the information in the Source of Truth.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">plugin</span><span class="pi">:</span> <span class="s">netbox.netbox.nb_inventory</span> <span class="na">api_endpoint</span><span class="pi">:</span> <span class="s">http://netbox03</span> <span class="na">validate_certs</span><span class="pi">:</span> <span class="no">false</span> <span class="na">config_context</span><span class="pi">:</span> <span class="no">false</span> <span class="na">group_by</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">device_roles</span> <span class="pi">-</span> <span class="s">platforms</span> <span class="na">compose</span><span class="pi">:</span> <span class="na">ansible_network_os</span><span class="pi">:</span> <span class="s">platform.slug</span> <span class="na">query_filters</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">site</span><span class="pi">:</span> <span class="s2">"</span><span class="s">minnesota01"</span> <span class="pi">-</span> <span class="na">has_primary_ip</span><span class="pi">:</span> <span class="s">True</span> </code></pre></div></div> <h2 id="extending-netbox-with-plugins">Extending NetBox with Plugins</h2> <p>One of the more recent feature additions to NetBox itself is the ability to extend it via your own or community driven plugins. From the wiki: “Plugins are packaged Django apps that can be installed alongside NetBox to provide custom functionality not present in the core application” <a href="https://github.com/netbox-community/netbox/wiki/Plugins">GitHub Link</a>. You can find some of the featured plugins in the community at that link. Some include:</p> <ul> <li><a href="https://github.com/sjm-steffann/netbox-ddns">Dynamic DNS Connector</a></li> <li><a href="https://github.com/networktocode/ntc-netbox-plugin-onboarding">NetBox Onboarding Plugin</a> (from Network to Code) - This will read additional information about the device and make updates to NetBox</li> <li><a href="https://github.com/k01ek/netbox-qrcode">NetBox QR Code</a> - Generate QR Codes about the device</li> <li><a href="https://github.com/jeremyschulman/netbox-plugin-auth-saml2">SSO using SAML2</a></li> </ul> <p>There are many plugins available to the community for you to choose from—or you can write your own add ons! <a href="https://github.com/topics/netbox-plugi">Search on GitHub</a> for the topic NetBox Plugin.</p> <h2 id="summary">Summary</h2> <p>NetBox and Ansible together are a great combination for your network automation needs!</p> <p>NetBox is an excellent open source tool that helps make it easy to create, update, and consume as a Source of Truth. The APIs are easy to use and make updates to the DB with, even if you did not want to use the NetBox Collection available for Ansible. Having a tool that is flexible, capable, and accurate is a must for delivering automation via a Source of Truth. NetBox delivers on each of these.</p> <p>This post was inspired by a presentation done in March 2020 at the Minneapolis Ansible Meetup. For additional material on this, I have many of these tasks available as a working example on <a href="https://github.com/jvanderaa/ansible_netbox_demo">GitHub</a>. The YouTube recording of the presentation from the <a href="https://www.youtube.com/watch?v=GyQf5F0gr3w">Ansible Meetup</a> is available.</p> <p>-Josh</p>Josh VanDeraaThis content was originally posted on the Ansible Blog on 2020-12-08. Here you will learn about NetBox at a high level, how it works to become a Source of Truth (SoT), and look into the use of the Ansible Content Collection, which is available on Ansible Galaxy. The goal is to show some of the capabilities that make NetBox a terrific tool and will be using NetBox as your network Source of Truth for automation!Parsing XML with Python and Ansible2020-12-21T00:00:00+00:002020-12-21T00:00:00+00:00https://blog.networktocode.com/post/parsing-xml-with-python-and-ansible<p>Which data type is more popular, JSON or XML? I believe the overwhelming majority will say “JSON”, which is understandable, because JSON is easier to read, easier to understand, and more human friendly. Besides that, there are a lot of reading materials around JSON and how to handle such data. But what to do when only XML is supported?</p> <p>The default solution for most of us today is to convert XML to JSON or Python dictionary. But this approach has some significant drawbacks because XML and JSON are not 100% compatible. XML doesn’t have the same distinction between a list and a dictionary. Depending on what you are processing you can end up with a different datastructure if you have one element returned, an interface for example, or multiple.</p> <p>There is a better way to process XML data in Python by using the native construct and libraries available. This blog is intended to give you tips and tricks on how to parse XML data with Python and Ansible.</p> <h2 id="xml-data-structure">XML Data Structure</h2> <p>In this section, we will briefly go over the XML Data Structure and the terminology used in it, which will help us better understand the XML data.</p> <p>The image below depicts a sample XML data in a tree format.</p> <p><img src="../../../static/images/blog_posts/parsing_xml_python_ansible/1.png" alt="XML Tree Structure" /></p> <p>The corresponding XML data will look like this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;rpc-reply&gt;</span> <span class="nt">&lt;interface-information&gt;</span> <span class="nt">&lt;physical-interface</span> <span class="na">type=</span><span class="s">'eth'</span><span class="nt">&gt;</span> <span class="nt">&lt;name&gt;</span>ge-0/0/0<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;admin-status&gt;</span>up<span class="nt">&lt;/admin-status&gt;</span> <span class="nt">&lt;oper-status&gt;</span>up<span class="nt">&lt;/oper-status&gt;</span> <span class="nt">&lt;logical-interface&gt;</span> <span class="nt">&lt;name&gt;</span>ge-0/0/0.0<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;admin-status&gt;</span>up<span class="nt">&lt;/admin-status&gt;</span> <span class="nt">&lt;oper-status&gt;</span>up<span class="nt">&lt;/oper-status&gt;</span> <span class="nt">&lt;filter-information&gt;</span> <span class="nt">&lt;/filter-information&gt;</span> <span class="nt">&lt;address-family&gt;</span> <span class="nt">&lt;address-family-name&gt;</span>inet<span class="nt">&lt;/address-family-name&gt;</span> <span class="nt">&lt;interface-address&gt;</span> <span class="nt">&lt;ifa-local&gt;</span>172.16.0.151/24<span class="nt">&lt;/ifa-local&gt;</span> <span class="nt">&lt;/interface-address&gt;</span> <span class="nt">&lt;/address-family&gt;</span> <span class="nt">&lt;/logical-interface&gt;</span> <span class="nt">&lt;/physical-interface&gt;</span> <span class="nt">&lt;/interface-information&gt;</span> <span class="nt">&lt;/rpc-reply&gt;</span> </code></pre></div></div> <blockquote> <p><strong>NOTE</strong> <br /> Some XML data will have namespaces, and we will address them a bit later.</p> </blockquote> <p>Now let’s explain the terminology depicted in the image:</p> <ol> <li><strong><em>root element</em></strong> - The element at the root of the XML tree. XML data will always have one root element. This element can also be referred to as parent element.</li> <li><strong><em>child element</em></strong> - Any element right below the root element is a child element. The root element may contain one or more child elements.</li> <li><strong><em>attribute</em></strong> - A tag for an element providing some information about it.</li> <li><strong><em>data</em></strong> - The data the element contains.</li> </ol> <blockquote> <p><strong>NOTE</strong> <br /> This is not a complete overview of the XML, but rather a quick introduction to it. The complete information about the XML can be found <a href="https://www.w3schools.com/xml/default.asp">here</a>.</p> </blockquote> <h2 id="parsing-xml-with-python">Parsing XML with Python</h2> <p>Python has a very sophisticated built-in library called <code class="highlighter-rouge">xml.etree.ElementTree</code> to deal with XML data. Before we jump into the Python interpreter and start parsing the data, we will need to address XML XPath and the methods available to the <code class="highlighter-rouge">xml.etree.ElementTree</code> class.</p> <h3 id="xml-xpath-support">XML XPath Support</h3> <p>XPath uses path expressions to find element(s) and relative data in the XML document. The <code class="highlighter-rouge">xml.etree.ElementTree</code> supports the following XPath expressions<sup>1</sup>:</p> <table> <thead> <tr> <th>Syntax</th> <th>Meaning</th> </tr> </thead> <tbody> <tr> <td>tag</td> <td>Selects all child elements with the given tag. For example, spam selects all child elements named spam, and spam/egg selects all grandchildren named egg in all children named spam. <code class="highlighter-rouge">{namespace}*</code> selects all tags in the given namespace, <code class="highlighter-rouge">{*}spam</code> selects tags named spam in any (or no) namespace, and <code class="highlighter-rouge">{}*</code> selects only tags that are not in a namespace.</td> </tr> <tr> <td>*</td> <td>Selects all child elements, including comments and processing instructions. For example, <code class="highlighter-rouge">*/egg</code> selects all grandchildren named egg.</td> </tr> <tr> <td>.</td> <td>Selects the current node. This is mostly useful at the beginning of the path, to indicate that it’s a relative path.</td> </tr> <tr> <td>//</td> <td>Selects all subelements, on all levels beneath the current element. For example, .//egg selects all egg elements in the entire tree.</td> </tr> <tr> <td>..</td> <td>Selects the parent element. Returns None if the path attempts to reach the ancestors of the start element (the element find was called on).</td> </tr> <tr> <td>[@attrib]</td> <td>Selects all elements that have the given attribute.</td> </tr> <tr> <td>[@attrib=’value’]</td> <td>Selects all elements for which the given attribute has the given value. The value cannot contain quotes.</td> </tr> <tr> <td>[tag]</td> <td>Selects all elements that have a child named tag. Only immediate children are supported.</td> </tr> <tr> <td>[.=’text’]</td> <td>Selects all elements whose complete text content, including descendants, equals the given <code class="highlighter-rouge">text</code>.</td> </tr> <tr> <td>[tag=’text’]</td> <td>Selects all elements that have a child named <code class="highlighter-rouge">tag</code> whose complete text content, including descendants, equals the given <code class="highlighter-rouge">text</code>.</td> </tr> <tr> <td>[position]</td> <td>Selects all elements that are located at the given position. The position can be either an integer (1 is the first position), the expression <code class="highlighter-rouge">last()</code> (for the last position), or a position relative to the last position (e.g., <code class="highlighter-rouge">last()-1)</code>.</td> </tr> </tbody> </table> <h3 id="xmletreeelementtree-methods-and-attributes">xml.etree.ElementTree Methods and Attributes</h3> <p>These are the methods and attributes that we will be using:</p> <blockquote> <p><strong>NOTE</strong><br /> All elements of the loaded XML data will be objects of the <code class="highlighter-rouge">Element</code> Python class and will support these methods and attributes.</p> </blockquote> <p><strong>Methods:</strong></p> <ul> <li><code class="highlighter-rouge">iter("&lt;element&gt;")</code> - recursively iterates over all the sub-tree below the root element. Requires the element name.</li> <li><code class="highlighter-rouge">findall("&lt;xpath-expression&gt;") </code>- finds elements using XPath expression.</li> <li><code class="highlighter-rouge">find("&lt;element&gt;")</code> - finds the first child element with a particular tag.</li> <li><code class="highlighter-rouge">get("&lt;attribute-tag&gt;")</code> - gets the element’s attribute value.</li> </ul> <p><strong>Attributes:</strong></p> <ul> <li><code class="highlighter-rouge">tag</code> - shows the tag name of the element (<code class="highlighter-rouge">rpc-reply</code>).</li> <li><code class="highlighter-rouge">attrib</code> - shows the value of the attribute (<code class="highlighter-rouge">{'type': 'eth'}</code>).</li> <li><code class="highlighter-rouge">text</code> - shows the data assigned to the element (<code class="highlighter-rouge">ge-0/0/0.0</code>).</li> </ul> <p><br /> Now we are ready to work with the XML data. Let’s start by opening the Python interpreter and importing the <code class="highlighter-rouge">xml.etree.ElementTree</code> class as <code class="highlighter-rouge">ET</code> for brevity.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">xml.etree.ElementTree</span> <span class="k">as</span> <span class="n">ET</span> </code></pre></div></div> <p>We will need to load the data now. It can be done two ways. Either load the XML file with the <code class="highlighter-rouge">parse()</code> method then obtain the root element with <code class="highlighter-rouge">getroot()</code> method, or load it from string using <code class="highlighter-rouge">fromstring()</code>.</p> <blockquote> <p><strong>NOTE</strong><br /> We will be using the XML data shown in the XML Data Structure section, but with more interfaces.</p> </blockquote> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">tree</span> <span class="o">=</span> <span class="n">ET</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">'interface_data.xml'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">root</span> <span class="o">=</span> <span class="n">tree</span><span class="o">.</span><span class="n">getroot</span><span class="p">()</span> </code></pre></div></div> <p>or</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">root</span> <span class="o">=</span> <span class="n">ET</span><span class="o">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">interface_data</span><span class="p">)</span> </code></pre></div></div> <p>To print the tag of the root element we can use the <code class="highlighter-rouge">tag</code> attribute</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">root</span><span class="o">.</span><span class="n">tag</span> <span class="s">'rpc-reply</span><span class="err"> </span></code></pre></div></div> <p>Let’s iterate of the child elements of the root element and print its tag names:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">root</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">child</span><span class="o">.</span><span class="n">tag</span><span class="p">)</span> <span class="o">...</span> <span class="n">interface</span><span class="o">-</span><span class="n">information</span> </code></pre></div></div> <p>As we can see from the output, the root element has only one child element, with <code class="highlighter-rouge">interface-information</code> tag name. To see more data, we can iterate over the <code class="highlighter-rouge">interface-information</code> element and print its elements’ tag names and attributes:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">root</span><span class="p">:</span> <span class="o">...</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">child</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">"Element Name: {element.tag} Element Attribute: {element.attrib}"</span><span class="p">)</span> <span class="o">...</span> <span class="n">Element</span> <span class="n">Name</span><span class="p">:</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="n">Element</span> <span class="n">Attribute</span><span class="p">:</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">Element</span> <span class="n">Name</span><span class="p">:</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="n">Element</span> <span class="n">Attribute</span><span class="p">:</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">Element</span> <span class="n">Name</span><span class="p">:</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="n">Element</span> <span class="n">Attribute</span><span class="p">:</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">Element</span> <span class="n">Name</span><span class="p">:</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="n">Element</span> <span class="n">Attribute</span><span class="p">:</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">Element</span> <span class="n">Name</span><span class="p">:</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="n">Element</span> <span class="n">Attribute</span><span class="p">:</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> </code></pre></div></div> <p>Even though the nested iteration works, it is not optimal, since the XML tree can have several layers. Instead, we can use <code class="highlighter-rouge">iter()</code> method and provide an element name as a parameter. In this case, we will iterate over only the found elements. For example, let’s iterate over all <code class="highlighter-rouge">name</code> elements and show their contents.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="nb">iter</span><span class="p">(</span><span class="s">'name'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mf">0.0</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">1</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mf">1.0</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">2</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mf">2.0</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">3</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mf">3.0</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">4</span> <span class="n">ge</span><span class="o">-</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mf">4.0</span> </code></pre></div></div> <p>In the same way, we can iterate over the elements in question, but with the help of the <code class="highlighter-rouge">findall()</code> method. For it, we will need to provide XPath expression as an argument. This time, we would like to print all IP addresses that are assigned to the <code class="highlighter-rouge">ifa-local</code> element:</p> <blockquote> <p><strong>NOTE</strong> <br /> The period before the forward slashes indicates that the search needs to happen from current element.</p> </blockquote> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//ifa-local'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="mf">172.16.0.151</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">10.10.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">172.31.177.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.0.1</span><span class="o">/</span><span class="mi">24</span> </code></pre></div></div> <p>The XPath expression to find a particular element with a specific attribute will be a little bit different. For it, we will need to use <code class="highlighter-rouge">findall(.//*[@type])</code> syntax to search from current element against all the elements that have an attribute tag of <code class="highlighter-rouge">type</code>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//*[@type]'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">tag</span><span class="p">,</span> <span class="n">element</span><span class="o">.</span><span class="n">attrib</span><span class="p">)</span> <span class="o">...</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> <span class="n">physical</span><span class="o">-</span><span class="n">interface</span> <span class="p">{</span><span class="s">'type'</span><span class="p">:</span> <span class="s">'eth'</span><span class="p">}</span> </code></pre></div></div> <h3 id="parsing-xml-data-ith-namespaces">Parsing XML Data ith Namespaces</h3> <p>XML Namespaces are used to group certain elements identified by Uniform Resource Identifier (URI) and avoid any element name conflicts in the XML document. Detailed information about XML Namespaces can be found <a href="https://www.w3.org/TR/REC-xml-names/">here</a>.</p> <p>Let’s look at an XML data with name spaces.</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;rpc-reply</span> <span class="na">xmlns:junos=</span><span class="s">"http://xml.juniper.net/junos/20.2R0/junos"</span><span class="nt">&gt;</span> <span class="nt">&lt;interface-information</span> <span class="na">xmlns=</span><span class="s">"http://xml.juniper.net/junos/20.2R0/junos-interface"</span> <span class="na">junos:style=</span><span class="s">"terse"</span><span class="nt">&gt;</span> <span class="nt">&lt;physical-interface</span> <span class="na">type=</span><span class="s">'eth'</span><span class="nt">&gt;</span> <span class="nt">&lt;name&gt;</span>ge-0/0/0<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;admin-status&gt;</span>up<span class="nt">&lt;/admin-status&gt;</span> <span class="nt">&lt;oper-status&gt;</span>up<span class="nt">&lt;/oper-status&gt;</span> <span class="nt">&lt;logical-interface&gt;</span> <span class="nt">&lt;name&gt;</span>ge-0/0/0.0<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;admin-status&gt;</span>up<span class="nt">&lt;/admin-status&gt;</span> <span class="nt">&lt;oper-status&gt;</span>up<span class="nt">&lt;/oper-status&gt;</span> <span class="nt">&lt;filter-information&gt;</span> <span class="nt">&lt;/filter-information&gt;</span> <span class="nt">&lt;address-family&gt;</span> <span class="nt">&lt;address-family-name&gt;</span>inet<span class="nt">&lt;/address-family-name&gt;</span> <span class="nt">&lt;interface-address&gt;</span> <span class="nt">&lt;ifa-local&gt;</span>172.16.0.151/24<span class="nt">&lt;/ifa-local&gt;</span> <span class="nt">&lt;/interface-address&gt;</span> <span class="nt">&lt;/address-family&gt;</span> <span class="nt">&lt;/logical-interface&gt;</span> <span class="nt">&lt;/physical-interface&gt;</span> <span class="nt">&lt;/interface-information&gt;</span> <span class="nt">&lt;/rpc-reply&gt;</span> </code></pre></div></div> <p>If we try to find a specific element using one of the methods that we discussed before, soon we will find out that no elements are matched.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">tree_xmlns</span> <span class="o">=</span> <span class="n">ET</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">"interface_data_with_xmlns.xml"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">root_xmlns</span> <span class="o">=</span> <span class="n">tree_xmlns</span><span class="o">.</span><span class="n">getroot</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root_xmlns</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//ifa-local'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>The reason is that each element of the <code class="highlighter-rouge">interface-information</code> child element is appended with <code class="highlighter-rouge">{URI}</code> tag. To see how it looks, we can print the tag name of the root’s child-element:</p> <blockquote> <p><strong>NOTE</strong> <br /> The root element does not have the URI tag appended since it has only XML name definition rather than assignment.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">root_xmlns</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">tag</span><span class="p">)</span> <span class="p">{</span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">xml</span><span class="o">.</span><span class="n">juniper</span><span class="o">.</span><span class="n">net</span><span class="o">/</span><span class="n">junos</span><span class="o">/</span><span class="mf">20.2</span><span class="n">R0</span><span class="o">/</span><span class="n">junos</span><span class="o">-</span><span class="n">interface</span><span class="p">}</span><span class="n">interface</span><span class="o">-</span><span class="n">information</span> <span class="o">&gt;&gt;</span> </code></pre></div> </div> <p>Now, if we change the syntax and search for the all <code class="highlighter-rouge">ifa-local</code> elements, we should get the desired result:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root_xmlns</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//{http://xml.juniper.net/junos/20.2R0/junos-interface}ifa-local'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="mf">172.16.0.151</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">10.10.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">172.31.177.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.0.1</span><span class="o">/</span><span class="mi">24</span> </code></pre></div> </div> <p>Python recommends a different way to search for elements with namespace. For that, we will need to create a dictionary and map a key to the namespace URI, then, using the desired class method, provide the dictionary as a second argument. The syntax looks like this:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;</span> <span class="n">ns</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">interface</span><span class="o">=</span><span class="s">"http://xml.juniper.net/junos/20.2R0/junos-interface"</span><span class="p">)</span> <span class="o">&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root_xmlns</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//interface:ifa-local'</span><span class="p">,</span> <span class="n">ns</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="mf">172.16.0.151</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">10.10.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">172.31.177.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.0.1</span><span class="o">/</span><span class="mi">24</span> </code></pre></div> </div> <p>In Python 3.8 and above, the namespace can be referenced with an asterisk character(<code class="highlighter-rouge">*</code>):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;</span> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">root_xmlns</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.//{*}ifa-local'</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">...</span> <span class="mf">172.16.0.151</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">10.10.10.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">172.31.177.1</span><span class="o">/</span><span class="mi">24</span> <span class="mf">192.168.0.1</span><span class="o">/</span><span class="mi">24</span> </code></pre></div> </div> </blockquote> <h2 id="parsing-xml-with-ansible">Parsing XML with Ansible</h2> <p>Parsing XML with Ansible can be done using two methods. The first method involves using <code class="highlighter-rouge">community.general.xml</code> module which is <a href="https://docs.ansible.com/ansible/latest/collections/community/general/xml_module.html">very well documented</a>. The second, more interesting, one using the <code class="highlighter-rouge">parse_xml</code> filter. Here we will be showing the second option.</p> <h3 id="parse_xml-ansible-filter"><code class="highlighter-rouge">parse_xml</code> Ansible Filter</h3> <p>To use the filter, we will need to create a specification file. The file has two parts: the first part defines the elements that need to be extracted from the XML data. The second part defines the variables and their values that will be available in the Ansible Playbook. We will start by looking at the first part.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">keys</span><span class="pi">:</span> <span class="na">result</span><span class="pi">:</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">top</span><span class="pi">:</span> <span class="s">interface-information/physical-interface</span> <span class="na">items</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">name</span> <span class="na">admin_status</span><span class="pi">:</span> <span class="s">admin-status</span> <span class="na">oper_status</span><span class="pi">:</span> <span class="s">oper-status</span> <span class="na">ifl_name</span><span class="pi">:</span> <span class="s">logical-interface/name</span> <span class="na">ip_addr</span><span class="pi">:</span> <span class="s">.//ifa-local</span> </code></pre></div></div> <p>Let’s break it all down:</p> <ul> <li><code class="highlighter-rouge">keys</code> - This is a predefined root level key name.</li> <li><code class="highlighter-rouge">result</code> - The name of this key can be set to anything.</li> <li><code class="highlighter-rouge">value</code> - This is a predefined key which holds a Jinja2 variable which will be mapped to a variable that will be defined in the second part of the specification file.</li> <li><code class="highlighter-rouge">top</code> - This key holds the path to the element that will be iterated over for data extraction. Note that the path starts from the child element not the root element.</li> <li><code class="highlighter-rouge">items</code> - This is a predefined key which will hold multiple items. <ul> <li><code class="highlighter-rouge">name</code> - Key that will hold the value that is assigned to the <code class="highlighter-rouge">&lt;name&gt;&lt;/name&gt;</code> element.</li> <li><code class="highlighter-rouge">admin_status</code> - Key that will hold the value that is assigned to the <code class="highlighter-rouge">&lt;admin-status&gt;&lt;/admin-status&gt;</code> element.</li> <li><code class="highlighter-rouge">oper_status</code> - Key that will hold the value that is assigned to the <code class="highlighter-rouge">&lt;oper-status&gt;&lt;/oper-status&gt;</code> element.</li> <li><code class="highlighter-rouge">ifl_name</code> - Key that will hold the value that is assigned to the <code class="highlighter-rouge">&lt;name&gt;&lt;/name&gt;</code> element which is a child elment of the <code class="highlighter-rouge">&lt;logical-interface&gt;&lt;/logical-interface&gt;</code> element.</li> <li><code class="highlighter-rouge">ip_addr</code> - Key that will hold the value that is assigned to the <code class="highlighter-rouge">&lt;ifa-local&gt;&lt;/ifa-local&gt;</code> element. Note that here we are using XPath expression to locate the desired element.</li> </ul> </li> </ul> <p>Now let’s look at the second part of the file and break it down:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">vars</span><span class="pi">:</span> <span class="na">dev_intf</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">admin_status</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">oper_status</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">ifl_name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">ip_addr</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> </code></pre></div></div> <ul> <li><code class="highlighter-rouge">vars</code> - This is a well-known Ansible key for defining variables.</li> <li><code class="highlighter-rouge">dev_intf</code> - The root key named by us that will hold the data extracted from the XML. <ul> <li><code class="highlighter-rouge">name</code> - Variable key that will hold the value assigned to the <code class="highlighter-rouge">name</code> key defined under items.</li> <li><code class="highlighter-rouge">admin_status</code> - Variable key that will hold the value assigned to the <code class="highlighter-rouge">admin_status</code> defined key under items.</li> <li><code class="highlighter-rouge">oper_status</code> - Variable key that will hold the value assigned to the <code class="highlighter-rouge">oper_status</code> defined key under items.</li> <li><code class="highlighter-rouge">ifl_name</code> - Variable key that will hold the value assigned to the <code class="highlighter-rouge">ifl_name</code> defined key under items.</li> </ul> </li> </ul> <p>The entire specs file looks like this:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">keys</span><span class="pi">:</span> <span class="na">result</span><span class="pi">:</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">top</span><span class="pi">:</span> <span class="s">interface-information/physical-interface</span> <span class="na">items</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">name</span> <span class="na">admin_status</span><span class="pi">:</span> <span class="s">admin-status</span> <span class="na">oper_status</span><span class="pi">:</span> <span class="s">oper-status</span> <span class="na">ifl_name</span><span class="pi">:</span> <span class="s">logical-interface/name</span> <span class="na">ip_addr</span><span class="pi">:</span> <span class="s">.//ifa-local</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">dev_intf</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">admin_status</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">oper_status</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">ifl_name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="na">ip_addr</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> </code></pre></div></div> <p>Now let’s look at the simple Ansible Playbook in which we will use the <code class="highlighter-rouge">parse_xml</code> filter.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">READ XML</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">local</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">parsed_xml_data</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">parsed_xml_data</span> </code></pre></div></div> <p>Finally, let’s run the Playbook against the XML data and see the output.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ansible-playbook pb_parse_xml.yml PLAY [READ XML] ******************************************************************************************************* TASK [set_fact] ******************************************************************************************************* ok: [localhost] TASK [debug] ********************************************************************************************************** ok: [localhost] =&gt; { "xml_data_parsed": { "result": [ { "admin_status": "up", "ifl_name": "ge-0/0/0.0", "ip_addr": "172.16.0.151/24", "name": "ge-0/0/0", "oper_status": "up" }, { "admin_status": "up", "ifl_name": "ge-0/0/1.0", "ip_addr": "192.168.10.1/24", "name": "ge-0/0/1", "oper_status": "up" }, { "admin_status": "up", "ifl_name": "ge-0/0/2.0", "ip_addr": "10.10.10.1/24", "name": "ge-0/0/2", "oper_status": "up" }, { "admin_status": "up", "ifl_name": "ge-0/0/3.0", "ip_addr": "172.31.177.1/24", "name": "ge-0/0/3", "oper_status": "up" }, { "admin_status": "up", "ifl_name": "ge-0/0/4.0", "ip_addr": "192.168.0.1/24", "name": "ge-0/0/4", "oper_status": "up" } ] } } PLAY RECAP ************************************************************************************************************ localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre></div></div> <p>I hope this post will help you understand how to parse XML data natively in Python and Ansible without converting it to JSON automatically.</p> <p>-Armen</p>Armen MartirosyanWhich data type is more popular, JSON or XML? I believe the overwhelming majority will say “JSON”, which is understandable, because JSON is easier to read, easier to understand, and more human friendly. Besides that, there are a lot of reading materials around JSON and how to handle such data. But what to do when only XML is supported?Ansible Constructed Inventory2020-12-11T00:00:00+00:002020-12-11T00:00:00+00:00https://blog.networktocode.com/post/ansible-constructed-inventory<p>Ansible’s inventory management system is one of the strongest features of the platform. The options for using different inventory sources make it quite extensible. Starting with a static inventory of ini or yaml files can be fine for a small deployment or proof-of-concept, but this only scales so far. To scale beyond that, you will likely need to use a dynamic inventory plugin or script. With all the plugins that are available today, it is often better to go with one designed to work with your dynamic inventory source, but keeping in mind, that plugins are limited in their capability as-is.</p> <p>In the case of the NetBox inventory plugin, there are several options out of the box by which to group hosts using different data points already available within the application (<a href="https://docs.ansible.com/ansible/latest/collections/netbox/netbox/nb_inventory_inventory.html">docs here</a>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sites site tenants tenant racks rack rack_group rack_role tags tag device_roles role device_types device_type manufacturers manufacturer platforms platform region cluster cluster_type cluster_group is_virtual services </code></pre></div></div> <p>These are great options for a lot of use cases, but what if there is a need to group hosts by something that is not supported natively? Well, any inventory plugin can be extended with the functionality of the Ansible <code class="highlighter-rouge">constructed</code> builtin (<a href="https://docs.ansible.com/ansible/devel/collections/ansible/builtin/constructed_inventory.html">docs here</a>). Custom groupings can be created, using either the <code class="highlighter-rouge">groups</code> (which is based on a boolean) or the <code class="highlighter-rouge">keyed_groups</code> (which is based on naming groups using Jinja2 logic) parameters to dynamically assign hosts into groups… that probably does not mean much right now, but this will be demonstrated shortly by using the <code class="highlighter-rouge">keyed_groups</code> option.</p> <h2 id="simple-example">Simple Example</h2> <p>As a simple example, perhaps it is necessary to group on the network OS, which is stored in NetBox under the <code class="highlighter-rouge">platform</code> key, but a different group name is needed. It is possible to use the <code class="highlighter-rouge">keyed_groups</code> parameter to create groups with a custom prefix, <code class="highlighter-rouge">network_os</code> in this example <code class="highlighter-rouge">netbox_inventory.yml</code> file:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">keyed_groups</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">platform</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_os"</span> <span class="na">separator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">_"</span> </code></pre></div></div> <p>With this configuration, since all the devices in this instance are Cisco IOS, they are in a single group, named <code class="highlighter-rouge">network_os_ios</code></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> brandomando@brandomando:<span class="nv">$ </span>ansible-inventory <span class="nt">-i</span> netbox_inventory.yml <span class="nt">--graph</span> @all: |--@network_os_ios: | |--LVO-RTR-01 | |--RVA-RTR-01 | |--LVO-SWI-01 | |--RVA-SWI-01 |--@ungrouped: </code></pre></div></div> <p>Here you can see that the result is equivalant to <code class="highlighter-rouge">'network_os' + '_' + platform</code> which dynamically created a group called <code class="highlighter-rouge">network_os_ios</code>.</p> <h2 id="slightly-more-complex-jinja-filter-example">Slightly More Complex Jinja Filter Example</h2> <p>This functionality can be extended with any Python code by building a custom filter. Let’s see what that looks like.</p> <p>One attribute of a device model that you may want to group on is device family, but that is not one of the natively available options, since this is not a field in NetBox. A simple filter can be used to map the <code class="highlighter-rouge">device_type</code> to a <code class="highlighter-rouge">device family</code>, which can then be used to dynamically assign hosts to their associated family group.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">FilterModule</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">filters</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="p">{</span> <span class="s">'devicetype_family'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">devicetype_family</span> <span class="p">}</span> <span class="k">def</span> <span class="nf">devicetype_family</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">device_type</span><span class="p">):</span> <span class="n">devicetype_map</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'ISR1921'</span><span class="p">:</span> <span class="s">'isr1k'</span><span class="p">,</span> <span class="s">'ISR1941'</span><span class="p">:</span> <span class="s">'isr1k'</span><span class="p">,</span> <span class="s">'ISR4321'</span><span class="p">:</span> <span class="s">'isr4k'</span><span class="p">,</span> <span class="s">'ISR4331'</span><span class="p">:</span> <span class="s">'isr4k'</span><span class="p">,</span> <span class="s">'ws-c3560cx-12pc-s'</span><span class="p">:</span> <span class="s">'cat3k'</span><span class="p">,</span> <span class="s">'ws-c3650-24ps'</span><span class="p">:</span> <span class="s">'cat3k'</span><span class="p">,</span> <span class="p">}</span> <span class="k">return</span> <span class="n">devicetype_map</span><span class="p">[</span><span class="n">device_type</span><span class="p">]</span> </code></pre></div></div> <p>In the NetBox <code class="highlighter-rouge">nb_inventory</code> plugin configuration file, we can use the <code class="highlighter-rouge">keyed_groups</code> functionality, in combination with the custom filter, to create a dynamic inventory group assignment.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">keyed_groups</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">device_type | devicetype_family</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">family"</span> <span class="na">separator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">_"</span> </code></pre></div></div> <p>By running the <code class="highlighter-rouge">keyed_groups</code> key through the <code class="highlighter-rouge">devicetype_family</code> filter, the Ansible dynamic inventory will now have custom groups that are dynamically assigned based on the <code class="highlighter-rouge">device_type</code> parameter returned by NetBox</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> brandomando@brandomando:<span class="nv">$ </span>ansible-inventory <span class="nt">-i</span> netbox_inventory.yml <span class="nt">--graph</span> @all: |--@family_cat3k: | |--LVO-SWI-01 | |--RVA-SWI-01 |--@family_isr1k: | |--LVO-RTR-01 |--@family_isr4k: | |--RVA-RTR-01 |--@role_rtr: | |--LVO-RTR-01 | |--RVA-RTR-01 |--@role_swi: | |--LVO-SWI-01 | |--RVA-SWI-01 |--@site_lvo: | |--LVO-RTR-01 | |--LVO-SWI-01 |--@site_rva: | |--RVA-RTR-01 | |--RVA-SWI-01 |--@ungrouped: </code></pre></div></div> <h2 id="compose-example">Compose Example</h2> <p>In addition to being able to group by custom logic, we can also use filters within the <code class="highlighter-rouge">compose</code> section of the NetBox inventory plugin config file to set custom variables in our rendered inventory as well. Let’s use the same filter plugin in this example.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">compose</span><span class="pi">:</span> <span class="na">ansible_network_os</span><span class="pi">:</span> <span class="s">platform.slug</span> <span class="na">family</span><span class="pi">:</span> <span class="s">device_type.slug | devicetype_family</span> </code></pre></div></div> <p>Looking at an ansible-inventory output for one of the hosts, we can see that <code class="highlighter-rouge">ansible_network_os</code> is now dynamically assigned using the <code class="highlighter-rouge">platform</code> parameter from NetBox, and the <code class="highlighter-rouge">family</code> variable is now set by our filter.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> brandomando@brandomando:<span class="nv">$ </span>ansible-inventory <span class="nt">-i</span> ./lib/netbox_inventory.yml <span class="nt">--host</span> LVO-RTR-01 <span class="nt">-y</span> ansible_host: 10.100.0.5 ansible_network_os: ios &lt;<span class="nt">---</span> generated from <span class="s2">"platform.slug"</span> custom_fields: <span class="o">{}</span> device_type: isr4331 family: isr4k &lt;<span class="nt">---</span> generated from <span class="s2">"device_type.slug | devicetype_family"</span> is_virtual: <span class="nb">false </span>manufacturer: cisco platform: ios primary_ip4: 10.100.0.5 regions: - las-vegas-nv - us-west - united-states - north-america role: rtr services: <span class="o">[]</span> site: lvo tags: <span class="o">[]</span> </code></pre></div></div> <p>These options are all features from the Ansible builtin <code class="highlighter-rouge">constructed</code> inventory plugin, which the NetBox community <code class="highlighter-rouge">netbox.netbox.nb_inventory</code> plugin extends.</p> <h2 id="full-example">Full Example</h2> <p>Demonstrating all of the components together:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">plugin</span><span class="pi">:</span> <span class="s">netbox.netbox.nb_inventory</span> <span class="na">api_endpoint</span><span class="pi">:</span> <span class="s">http://localhost:8000</span> <span class="na">validate_certs</span><span class="pi">:</span> <span class="s">True</span> <span class="na">config_context</span><span class="pi">:</span> <span class="s">False</span> <span class="na">group_by</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">device_roles</span> <span class="na">device_query_filters</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">has_primary_ip</span><span class="pi">:</span> <span class="s1">'</span><span class="s">true'</span> <span class="na">keyed_groups</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">platform</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_os"</span> <span class="na">separator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">_"</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">device_type | devicetype_family</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">family"</span> <span class="na">separator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">_"</span> <span class="na">compose</span><span class="pi">:</span> <span class="na">ansible_network_os</span><span class="pi">:</span> <span class="s">platform.slug</span> <span class="na">family</span><span class="pi">:</span> <span class="s">device_type.slug | devicetype_family</span> </code></pre></div></div> <h2 id="recap">Recap</h2> <p>This functionality is great for both users of the plugin and the plugin writers. The plugin writer no longer has to consider every possible use case that a user might have. The user can leverage the extensibility that is provided by the Ansible plugin framework and can be used for any number of customized groupings or variable assignments within dynamic inventories.</p> <p>-Brandon</p>Brandon Donohoe (guest post)Ansible’s inventory management system is one of the strongest features of the platform. The options for using different inventory sources make it quite extensible. Starting with a static inventory of ini or yaml files can be fine for a small deployment or proof-of-concept, but this only scales so far. To scale beyond that, you will likely need to use a dynamic inventory plugin or script. With all the plugins that are available today, it is often better to go with one designed to work with your dynamic inventory source, but keeping in mind, that plugins are limited in their capability as-is.Parsing Strategies - PyATS Genie Parsers2020-12-01T00:00:00+00:002020-12-01T00:00:00+00:00https://blog.networktocode.com/post/parsing-strategies-pyats-genie<p>Thank you for joining me for Part 3 of the <a href="http://blog.networktocode.com/post/parsing-strategies-intro/">parsing strategies</a> blog series. This post will dive deeper into using <a href="https://developer.cisco.com/docs/pyats/#!introduction">Cisco’s PyATS Genie library</a> for parsing. For further reference in this blog post, I’ll be referring to the Genie library just by Genie. Genie also uses Regular Expressions (RegEx) under the hood to parse the output received from a device whether it’s semi-structured data, XML, YANG, etc. This is a key difference from other parsers, but for now, let’s stick with parsing semi-structured data.</p> <p>Let’s move on and dive deeper into what the <strong>show lldp neighbors</strong> parser looks like, how it works, and how we need to modify our existing playbook to use the Genie parsers.</p> <h2 id="genie-primer">Genie Primer</h2> <p>Before we get too deep into how Genie works, you can see all the <a href="https://developer.cisco.com/docs/genie-docs/">available parsers</a>. The number of parsers has increased dramatically over the last several months and is starting to include more vendors, which is great to see.</p> <p>Genie uses Python classes to build two important parsing functions:</p> <ul> <li><strong>Schema class</strong>: This class defines the schema the structured output should adhere to.</li> <li><strong>Parser class</strong>: This class defines the actual parsing methods for the specific command.</li> </ul> <p>One key difference between what we’ve covered so far and Genie parsers is the ability to connect to devices and grab the necessary output, which is the default behavior, but also allows users to provide the output instead, thus working like most parsers that are separate from the device interaction.</p> <p>Another key difference is the ability to use other parsing strategies within Genie such as <strong>TextFSM</strong> or <strong>Template Text Parser (TTP)</strong>, but for the sake of this post, we will be covering RegEx.</p> <blockquote> <p>You can find more detailed information on how the Genie parsers work at their <a href="https://developer.cisco.com/docs/pyats-development-guide/">developer guide</a>.</p> </blockquote> <p>Let’s dive into our particular parser.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">"""show_lldp.py supported commands: * show lldp * show lldp entry * * show lldp entry [&lt;WORD&gt;] * show lldp interface [&lt;WORD&gt;] * show lldp neighbors * show lldp neighbors detail * show lldp traffic """</span> <span class="kn">import</span> <span class="nn">re</span> <span class="kn">from</span> <span class="nn">genie.metaparser</span> <span class="kn">import</span> <span class="n">MetaParser</span> <span class="kn">from</span> <span class="nn">genie.metaparser.util.schemaengine</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> \ <span class="n">Any</span><span class="p">,</span> \ <span class="n">Optional</span><span class="p">,</span> \ <span class="n">Or</span><span class="p">,</span> \ <span class="n">And</span><span class="p">,</span> \ <span class="n">Default</span><span class="p">,</span> \ <span class="n">Use</span> <span class="c1"># import parser utils </span><span class="kn">from</span> <span class="nn">genie.libs.parser.utils.common</span> <span class="kn">import</span> <span class="n">Common</span> </code></pre></div></div> <p>We can see this parser is declared in the <code class="highlighter-rouge">show_lldp.py</code> module and supports several variations of the <strong>show lldp</strong> commands. It then imports <code class="highlighter-rouge">re</code>, which is the built in RegEx library. The next import is the <code class="highlighter-rouge">MetaParser</code> that makes sure that the parsers’ output adheres to the defined schema. After <code class="highlighter-rouge">MetaParser</code> is imported, the schema related imports take place that helps build the actual schema we’ll see shortly. After that, the <a href="https://github.com/CiscoTestAutomation/genieparser/blob/95eb4aa2eaa84e142f9b384903fcfe878502244e/src/genie/libs/parser/utils/common.py#L471">Common</a> class that provides helper functions is imported.</p> <p>The schema class provides us insight into what the output will look like. Let’s take a look at the initial definition.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ShowLldpNeighborsSchema</span><span class="p">(</span><span class="n">MetaParser</span><span class="p">):</span> <span class="s">""" Schema for show lldp neighbors """</span> <span class="n">schema</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'total_entries'</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="s">'interfaces'</span><span class="p">:</span> <span class="p">{</span> <span class="n">Any</span><span class="p">():</span> <span class="p">{</span> <span class="s">'port_id'</span><span class="p">:</span> <span class="p">{</span> <span class="n">Any</span><span class="p">():</span> <span class="p">{</span> <span class="s">'neighbors'</span><span class="p">:</span> <span class="p">{</span> <span class="n">Any</span><span class="p">():</span> <span class="p">{</span> <span class="s">'hold_time'</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">Optional</span><span class="p">(</span><span class="s">'capabilities'</span><span class="p">):</span> <span class="nb">list</span><span class="p">,</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>We can see that <code class="highlighter-rouge">ShowLldpNeighborsSchema</code> will be a subclass of the <code class="highlighter-rouge">MetaParser</code> class imported at the beginning of the file. Within the <code class="highlighter-rouge">ShowLldpNeighborsSchema</code> class, we define our <code class="highlighter-rouge">schema</code> attribute used to make sure our output adheres to the schema before returning it to the user.</p> <p>The schema is a dictionary and is expecting a <code class="highlighter-rouge">total_entries</code> key with an integer value and an <code class="highlighter-rouge">interfaces</code> key, used to define a dictionary. Each interface will be a key within the <code class="highlighter-rouge">interfaces</code> dictionary and the data obtained from the output is defined in several other nested dictionaries. Each key value pair specifies the key and the <code class="highlighter-rouge">type</code> of value it must be. There are also <code class="highlighter-rouge">Optional</code> keys not required to pass schema validation.</p> <p>Now that we see the schema, we can mock up what our potential output would be.</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"total_entries"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="nl">"interfaces"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"GigabitEthernet1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"port_id"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"Gi1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"neighbors"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"iosv-0"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"capabilities"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"R"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"hold_time"</span><span class="p">:</span><span class="w"> </span><span class="mi">120</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Let’s move onto the defined parser class.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ShowLldpNeighbors</span><span class="p">(</span><span class="n">ShowLldpNeighborsSchema</span><span class="p">):</span> <span class="s">""" Parser for show lldp neighbors """</span> <span class="n">CAPABILITY_CODES</span> <span class="o">=</span> <span class="p">{</span><span class="s">'R'</span><span class="p">:</span> <span class="s">'router'</span><span class="p">,</span> <span class="s">'B'</span><span class="p">:</span> <span class="s">'mac_bridge'</span><span class="p">,</span> <span class="s">'T'</span><span class="p">:</span> <span class="s">'telephone'</span><span class="p">,</span> <span class="s">'C'</span><span class="p">:</span> <span class="s">'docsis_cable_device'</span><span class="p">,</span> <span class="s">'W'</span><span class="p">:</span> <span class="s">'wlan_access_point'</span><span class="p">,</span> <span class="s">'P'</span><span class="p">:</span> <span class="s">'repeater'</span><span class="p">,</span> <span class="s">'S'</span><span class="p">:</span> <span class="s">'station_only'</span><span class="p">,</span> <span class="s">'O'</span><span class="p">:</span> <span class="s">'other'</span><span class="p">}</span> <span class="n">cli_command</span> <span class="o">=</span> <span class="p">[</span><span class="s">'show lldp neighbors'</span><span class="p">]</span> </code></pre></div></div> <p>We can see the <code class="highlighter-rouge">ShowLldpNeighbors</code> class is inheriting the <code class="highlighter-rouge">ShowLldpNeighborsSchema</code> class we just covered. Now there is a mapping for the short form capabilities codes returned within the output when neighbors exist, and the long form the parser wants to return to the user.</p> <p>The next defined variable is <code class="highlighter-rouge">cli_command</code>. It specifies the commands executed by the parser if no output is provided.</p> <p>Let’s take look at the <code class="highlighter-rouge">cli</code> method to see what will be executed when a user specifies the <code class="highlighter-rouge">cli</code> parser for <code class="highlighter-rouge">show lldp neighbors</code> command.</p> <blockquote> <p>Each type of output will be specified as a method under the parser class. For example, if the device returns xml, there will be an <code class="highlighter-rouge">xml</code> method that will parse and return structured data that adheres to the same schema as the <code class="highlighter-rouge">cli</code> output.</p> </blockquote> <p>Let’s explore the code in bite size chunks to show what’s happening.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">cli</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">output</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="k">if</span> <span class="n">output</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span> <span class="n">cmd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli_command</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">out</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">device</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="n">out</span> <span class="o">=</span> <span class="n">output</span> <span class="n">parsed_output</span> <span class="o">=</span> <span class="p">{}</span> </code></pre></div></div> <p>We can see the <code class="highlighter-rouge">cli</code> method takes an optional argument named <code class="highlighter-rouge">output</code>, but defaults to <code class="highlighter-rouge">None</code>. The first logic determines whether the user has provided the <code class="highlighter-rouge">output</code> or whether the parser needs to execute the command against the device. The connection the parser uses is provided by the PyATS library. This means no other library is required such as <code class="highlighter-rouge">netmiko</code> or <code class="highlighter-rouge">napalm</code> to connect to the devices.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># Total entries displayed: 4 </span> <span class="n">p1</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="s">r'^Total\s+entries\s+displayed:\s+(?P&lt;entry&gt;\d+)$'</span><span class="p">)</span> <span class="c1"># Device ID Local Intf Hold-time Capability Port ID </span> <span class="c1"># router Gi1/0/52 117 R Gi0/0/0 </span> <span class="c1"># 10.10.191.107 Gi1/0/14 155 B,T 7038.eeff.572d </span> <span class="c1"># d89e.f3ff.58fe Gi1/0/33 3070 d89e.f3ff.58fe </span> <span class="n">p2</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="s">r'(?P&lt;device_id&gt;\S+)\s+(?P&lt;interfaces&gt;\S+)'</span> <span class="s">r'\s+(?P&lt;hold_time&gt;\d+)\s+(?P&lt;capabilities&gt;[A-Z,]+)?'</span> <span class="s">r'\s+(?P&lt;port_id&gt;\S+)'</span><span class="p">)</span> </code></pre></div></div> <p>After the <code class="highlighter-rouge">parsed_output</code> variable is instantiated the next step is to define the RegEx expressions used to find the valuable data within the device output. Since the output is tabulated, which means it’s defined as a table, all values we care about will be on the same line (row) for each neighbor.</p> <blockquote> <p>The parser uses <code class="highlighter-rouge">re.compile</code> to specify the RegEx expression ahead of time for use later in the code. Typically, <code class="highlighter-rouge">re.compile</code> is used when the same expression is used multiple times.</p> </blockquote> <p><code class="highlighter-rouge">p1</code> will provide the <code class="highlighter-rouge">total_entries</code> within our schema, by using the <strong>Named Capturing Groups</strong> ability within the <code class="highlighter-rouge">re</code> library. Luckily, Cisco provide great documentation within the code to tell you what each RegEx is expecting to capture. <code class="highlighter-rouge">p2</code> defines the RegEx used to capture the neighbor related information. We can see it uses mostly <code class="highlighter-rouge">\S+</code> which captures any non-whitespace since the output is straight forward. But we can see the <strong>capabilities</strong> named capture group is a bit more complicated. It’s expecting at least one or more capital letter or comma, and then zero or one of that RegEx expression. This may be better explained by their example if we look at the capabilities column, it shows that it can capture a single capability, two capabilities with a comma, or zero capabilities.</p> <p>Now let’s look at the remaining code to see how it uses these RegEx expressions.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">out</span><span class="o">.</span><span class="n">splitlines</span><span class="p">():</span> <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="c1"># Total entries displayed: 4 </span> <span class="n">m</span> <span class="o">=</span> <span class="n">p1</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">if</span> <span class="n">m</span><span class="p">:</span> <span class="n">parsed_output</span><span class="p">[</span><span class="s">'total_entries'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">m</span><span class="o">.</span><span class="n">groupdict</span><span class="p">()[</span><span class="s">'entry'</span><span class="p">])</span> <span class="k">continue</span> <span class="c1"># Device ID Local Intf Hold-time Capability Port ID </span> <span class="c1"># router Gi1/0/52 117 R Gi0/0/0 </span> <span class="c1"># 10.10.191.107 Gi1/0/14 155 B,T 7038.eeff.572d </span> <span class="c1"># d89e.f3ff.58fe Gi1/0/33 3070 d89e.f3ff.58fe </span> <span class="n">m</span> <span class="o">=</span> <span class="n">p2</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="k">if</span> <span class="n">m</span><span class="p">:</span> <span class="n">group</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">groupdict</span><span class="p">()</span> <span class="n">intf</span> <span class="o">=</span> <span class="n">Common</span><span class="o">.</span><span class="n">convert_intf_name</span><span class="p">(</span><span class="n">group</span><span class="p">[</span><span class="s">'interfaces'</span><span class="p">])</span> <span class="n">device_dict</span> <span class="o">=</span> <span class="n">parsed_output</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="s">'interfaces'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span> \ <span class="n">setdefault</span><span class="p">(</span><span class="n">intf</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span> \ <span class="n">setdefault</span><span class="p">(</span><span class="s">'port_id'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span> \ <span class="n">setdefault</span><span class="p">(</span><span class="n">group</span><span class="p">[</span><span class="s">'port_id'</span><span class="p">],</span> <span class="p">{})</span><span class="o">.</span>\ <span class="n">setdefault</span><span class="p">(</span><span class="s">'neighbors'</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span> \ <span class="n">setdefault</span><span class="p">(</span><span class="n">group</span><span class="p">[</span><span class="s">'device_id'</span><span class="p">],</span> <span class="p">{})</span> <span class="n">device_dict</span><span class="p">[</span><span class="s">'hold_time'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">group</span><span class="p">[</span><span class="s">'hold_time'</span><span class="p">])</span> <span class="k">if</span> <span class="n">group</span><span class="p">[</span><span class="s">'capabilities'</span><span class="p">]:</span> <span class="n">capabilities</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">(),</span> <span class="n">group</span><span class="p">[</span><span class="s">'capabilities'</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">','</span><span class="p">)))</span> <span class="n">device_dict</span><span class="p">[</span><span class="s">'capabilities'</span><span class="p">]</span> <span class="o">=</span> <span class="n">capabilities</span> <span class="k">continue</span> <span class="k">return</span> <span class="n">parsed_output</span> </code></pre></div></div> <p>We can see the parser performs a <code class="highlighter-rouge">for</code> loop through the output using the <code class="highlighter-rouge">splitlines</code> method to provide a list of each line within the output. It will strip any whitespace on either side of the string.</p> <p>The parser will attempt to match the <code class="highlighter-rouge">p1</code> compiled RegEx and if it captures it, will then add <code class="highlighter-rouge">total_entries</code> to the <code class="highlighter-rouge">parsed_output</code> dictionary and then <code class="highlighter-rouge">continue</code> to the next line in the output.</p> <p>If the parser didn’t capture anything for <code class="highlighter-rouge">p1</code>, it will then attempt to match <code class="highlighter-rouge">p2</code>. If a match occurs, it will then the <code class="highlighter-rouge">groupdict()</code> method to return all the named capture groups and their values as a dictionary.</p> <p>We can now see the parser uses the <code class="highlighter-rouge">convert_intf_name</code> method from the <code class="highlighter-rouge">Common</code> class imported at the top of the file.</p> <blockquote> <p>You can review the code <a href="https://github.com/CiscoTestAutomation/genieparser/blob/95eb4aa2eaa84e142f9b384903fcfe878502244e/src/genie/libs/parser/utils/common.py#L484">here</a>.</p> </blockquote> <p>Once the interface name is converted, the parser adds the <code class="highlighter-rouge">interfaces</code> dictionary to the <code class="highlighter-rouge">parsed_output</code> variable by extracting the information captured or defaulting to an empty dictionary for any non-captured data and then assigning it to the <code class="highlighter-rouge">device_dict</code> variable.</p> <p>After the <code class="highlighter-rouge">device_dict</code> is specified, it adds the hold time to it.</p> <p>The next step is to strip and split the capabilities into a list and add to the <code class="highlighter-rouge">device_dict</code> variable. The parser will then continue to the next line of the output.</p> <p>Once all lines are parsed, it will return the <code class="highlighter-rouge">parsed_output</code> to the user as long as it passes schema validation.</p> <p>I believe this can be easier to understand than something like TextFSM since it’s written in Python and Python is a popular language among network automation engineers.</p> <p>Let’s move on and review the topology again.</p> <h2 id="the-topology-again">The Topology.. Again</h2> <p>Below is a picture of the lab topology we’re using to validate <strong>LLDP neighbors</strong>. It’s a simple topology with three Cisco IOS routers connected together and have LLDP enabled.</p> <p><img src="../../../static/images/parsing/BlogPostTopology.png" alt="Topology" /></p> <h2 id="ansible-setup-again">Ansible Setup.. Again</h2> <p>We’ve already covered most of the Ansible setup in <a href="http://blog.networktocode.com/post/parsing-strategies-ntc-templates/">part 2</a>, but we’ll explain the small changes we have to make to use the Genie parsers within Ansible.</p> <p>Here is a look at a host var we’ve defined as a refresher since there are no changes here.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">approved_neighbors</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">local_intf</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span> <span class="na">neighbor</span><span class="pi">:</span> <span class="s2">"</span><span class="s">iosv-1"</span> <span class="pi">-</span> <span class="na">local_intf</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="na">neighbor</span><span class="pi">:</span> <span class="s2">"</span><span class="s">iosv-2"</span> </code></pre></div></div> <p>Now let’s take a look at the changes in <code class="highlighter-rouge">pb.validate.neighbors.yml</code>.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ios"</span> <span class="na">connection</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ansible.netcommon.network_cli"</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">no"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PARSE</span><span class="nv"> </span><span class="s">LLDP</span><span class="nv"> </span><span class="s">INFO</span><span class="nv"> </span><span class="s">INTO</span><span class="nv"> </span><span class="s">STRUCTURED</span><span class="nv"> </span><span class="s">DATA"</span> <span class="s">ansible.netcommon.cli_parse</span><span class="pi">:</span> <span class="na">command</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">lldp</span><span class="nv"> </span><span class="s">neighbors"</span> <span class="na">parser</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ansible.netcommon.pyats</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">lldp_neighbors"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">MANIPULATE</span><span class="nv"> </span><span class="s">THE</span><span class="nv"> </span><span class="s">DATA</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">BE</span><span class="nv"> </span><span class="s">IN</span><span class="nv"> </span><span class="s">THE</span><span class="nv"> </span><span class="s">SAME</span><span class="nv"> </span><span class="s">FORMAT</span><span class="nv"> </span><span class="s">AS</span><span class="nv"> </span><span class="s">TEXTFSM</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">PREVENT</span><span class="nv"> </span><span class="s">CHANGING</span><span class="nv"> </span><span class="s">FINAL</span><span class="nv"> </span><span class="s">ASSERTION</span><span class="nv"> </span><span class="s">TASK"</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">lldp_neighbors</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lldp_neighbors</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">convert_data</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ASSERT</span><span class="nv"> </span><span class="s">THE</span><span class="nv"> </span><span class="s">CORRECT</span><span class="nv"> </span><span class="s">NEIGHBORS</span><span class="nv"> </span><span class="s">ARE</span><span class="nv"> </span><span class="s">SEEN"</span> <span class="na">assert</span><span class="pi">:</span> <span class="na">that</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">lldp_neighbors</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">selectattr('local_interface',</span><span class="nv"> </span><span class="s">'equalto',</span><span class="nv"> </span><span class="s">item['local_intf'])</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">map(attribute='neighbor')</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">first</span><span class="nv"> </span><span class="s">==</span><span class="nv"> </span><span class="s">item['neighbor']"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">approved_neighbors</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <blockquote> <p>Using the <code class="highlighter-rouge">ansible.netcommon.pyats</code> parser requires <code class="highlighter-rouge">genie</code> and <code class="highlighter-rouge">pyats</code> to be installed via <code class="highlighter-rouge">pip install genie pyats</code>.</p> </blockquote> <p>There are a few things to dissect with the playbook. First, we changed the parser to <code class="highlighter-rouge">ansible.netcommon.pyats</code>. Second, we added another task to manipulate the data we get back from the parser, into a similar format as the second blog post in this series so we don’t have to change the last task. I did the conversion within a custom filter plugin due to the structure of the data and the ease of handling this within Python. You will see the output below once we run our playbook.</p> <h2 id="playbook-output">Playbook Output</h2> <p>Let’s go ahead and run the playbook and see what output we get.</p> <p><img src="../../../static/images/parsing/parsing-ansible-pyats.gif" alt="playbook-gif" /></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ ansible-playbook pb.validate.neighbors.yml <span class="nt">-k</span> <span class="nt">-vv</span> ansible-playbook 2.10.3 config file <span class="o">=</span> /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg configured module search path <span class="o">=</span> <span class="o">[</span><span class="s1">'/Users/myohman/.ansible/plugins/modules'</span>, <span class="s1">'/usr/share/ansible/plugins/modules'</span><span class="o">]</span> ansible python module location <span class="o">=</span> /Users/myohman/.virtualenvs/3.8/main/lib/python3.8/site-packages/ansible executable location <span class="o">=</span> /Users/myohman/.virtualenvs/3.8/main/bin/ansible-playbook python version <span class="o">=</span> 3.8.6 <span class="o">(</span>default, Nov 17 2020, 18:43:06<span class="o">)</span> <span class="o">[</span>Clang 12.0.0 <span class="o">(</span>clang-1200.0.32.27<span class="o">)]</span> Using /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg as config file SSH password: redirecting <span class="o">(</span><span class="nb">type</span>: callback<span class="o">)</span> ansible.builtin.yaml to community.general.yaml redirecting <span class="o">(</span><span class="nb">type</span>: callback<span class="o">)</span> ansible.builtin.yaml to community.general.yaml Skipping callback <span class="s1">'default'</span>, as we already have a stdout callback. Skipping callback <span class="s1">'minimal'</span>, as we already have a stdout callback. Skipping callback <span class="s1">'oneline'</span>, as we already have a stdout callback. PLAYBOOK: pb.validate.neighbors.yml <span class="k">***********************************************************************</span> 1 plays <span class="k">in </span>pb.validate.neighbors.yml PLAY <span class="o">[</span>ios] <span class="k">************************************************************************************************</span> META: ran handlers TASK <span class="o">[</span>Parse LLDP info into structured data] <span class="k">***************************************************************</span> task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:9 ok: <span class="o">[</span>iosv-2] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: interfaces: GigabitEthernet0/0: port_id: Gi0/1: neighbors: iosv-0: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/1: neighbors: iosv-1: capabilities: - R hold_time: 120 total_entries: 2 parsed: interfaces: GigabitEthernet0/0: port_id: Gi0/1: neighbors: iosv-0: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/1: neighbors: iosv-1: capabilities: - R hold_time: 120 total_entries: 2 stdout: |- Capability codes: <span class="o">(</span>R<span class="o">)</span> Router, <span class="o">(</span>B<span class="o">)</span> Bridge, <span class="o">(</span>T<span class="o">)</span> Telephone, <span class="o">(</span>C<span class="o">)</span> DOCSIS Cable Device <span class="o">(</span>W<span class="o">)</span> WLAN Access Point, <span class="o">(</span>P<span class="o">)</span> Repeater, <span class="o">(</span>S<span class="o">)</span> Station, <span class="o">(</span>O<span class="o">)</span> Other Device ID Local Intf Hold-time Capability Port ID iosv-1 Gi0/1 120 R Gi0/1 iosv-0 Gi0/0 120 R Gi0/1 Total entries displayed: 2 stdout_lines: &lt;omitted&gt; ok: <span class="o">[</span>iosv-0] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: interfaces: GigabitEthernet0/0: port_id: Gi0/0: neighbors: iosv-1: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/0: neighbors: iosv-2: capabilities: - R hold_time: 120 total_entries: 2 parsed: interfaces: GigabitEthernet0/0: port_id: Gi0/0: neighbors: iosv-1: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/0: neighbors: iosv-2: capabilities: - R hold_time: 120 total_entries: 2 stdout: |- Capability codes: <span class="o">(</span>R<span class="o">)</span> Router, <span class="o">(</span>B<span class="o">)</span> Bridge, <span class="o">(</span>T<span class="o">)</span> Telephone, <span class="o">(</span>C<span class="o">)</span> DOCSIS Cable Device <span class="o">(</span>W<span class="o">)</span> WLAN Access Point, <span class="o">(</span>P<span class="o">)</span> Repeater, <span class="o">(</span>S<span class="o">)</span> Station, <span class="o">(</span>O<span class="o">)</span> Other Device ID Local Intf Hold-time Capability Port ID iosv-2 Gi0/1 120 R Gi0/0 iosv-1 Gi0/0 120 R Gi0/0 Total entries displayed: 2 stdout_lines: &lt;omitted&gt; ok: <span class="o">[</span>iosv-1] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: interfaces: GigabitEthernet0/0: port_id: Gi0/0: neighbors: iosv-0: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/1: neighbors: iosv-2: capabilities: - R hold_time: 120 total_entries: 2 parsed: interfaces: GigabitEthernet0/0: port_id: Gi0/0: neighbors: iosv-0: capabilities: - R hold_time: 120 GigabitEthernet0/1: port_id: Gi0/1: neighbors: iosv-2: capabilities: - R hold_time: 120 total_entries: 2 stdout: |- Capability codes: <span class="o">(</span>R<span class="o">)</span> Router, <span class="o">(</span>B<span class="o">)</span> Bridge, <span class="o">(</span>T<span class="o">)</span> Telephone, <span class="o">(</span>C<span class="o">)</span> DOCSIS Cable Device <span class="o">(</span>W<span class="o">)</span> WLAN Access Point, <span class="o">(</span>P<span class="o">)</span> Repeater, <span class="o">(</span>S<span class="o">)</span> Station, <span class="o">(</span>O<span class="o">)</span> Other Device ID Local Intf Hold-time Capability Port ID iosv-2 Gi0/1 120 R Gi0/1 iosv-0 Gi0/0 120 R Gi0/0 Total entries displayed: 2 stdout_lines: &lt;omitted&gt; TASK <span class="o">[</span>MANIPULATE THE DATA TO BE IN STANDARD FORMAT] <span class="k">*******************************************************</span> task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:16 ok: <span class="o">[</span>iosv-0] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: - local_interface: Gi0/1 neighbor: iosv-2 - local_interface: Gi0/0 neighbor: iosv-1 ok: <span class="o">[</span>iosv-1] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: - local_interface: Gi0/1 neighbor: iosv-2 - local_interface: Gi0/0 neighbor: iosv-0 ok: <span class="o">[</span>iosv-2] <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_facts: lldp_neighbors: - local_interface: Gi0/1 neighbor: iosv-1 - local_interface: Gi0/0 neighbor: iosv-0 TASK <span class="o">[</span>Assert the correct neighbors are seen] <span class="k">**************************************************************</span> task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:20 ok: <span class="o">[</span>iosv-0] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/0'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-1'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/0 neighbor: iosv-1 msg: All assertions passed ok: <span class="o">[</span>iosv-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/0'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-0'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/0 neighbor: iosv-0 msg: All assertions passed ok: <span class="o">[</span>iosv-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/0'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-0'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/0 neighbor: iosv-0 msg: All assertions passed ok: <span class="o">[</span>iosv-0] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/1'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-2'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/1 neighbor: iosv-2 msg: All assertions passed ok: <span class="o">[</span>iosv-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/1'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-2'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/1 neighbor: iosv-2 msg: All assertions passed ok: <span class="o">[</span>iosv-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">={</span><span class="s1">'local_intf'</span>: <span class="s1">'Gi0/1'</span>, <span class="s1">'neighbor'</span>: <span class="s1">'iosv-1'</span><span class="o">})</span> <span class="o">=&gt;</span> <span class="nv">changed</span><span class="o">=</span><span class="nb">false </span>ansible_loop_var: item item: local_intf: Gi0/1 neighbor: iosv-1 msg: All assertions passed META: ran handlers META: ran handlers PLAY RECAP <span class="k">************************************************************************************************</span> iosv-0 : <span class="nv">ok</span><span class="o">=</span>3 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 iosv-1 : <span class="nv">ok</span><span class="o">=</span>3 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 iosv-2 : <span class="nv">ok</span><span class="o">=</span>3 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>I ran this playbook with some verbosity to show what each task returns and the format of our parsed data.</p> <p>If we take a closer look at the output of the first task, we can see under the <code class="highlighter-rouge">parsed</code> key as well as setting the fact (<code class="highlighter-rouge">lldp_neighbors</code>), that we have our structured data from running the raw output through Genie.</p> <p>The second task shows the loop for each host and the <code class="highlighter-rouge">item</code> it’s using during the loop. If you look back at our playbook, we’re using both the <code class="highlighter-rouge">local_intf</code> and <code class="highlighter-rouge">neighbor</code> for our assertions from our <code class="highlighter-rouge">approved_neighbors</code> variable.</p> <h2 id="summary">Summary</h2> <p>We converted the data using a custom filter plugin, but we could have easily adjusted the facts and final assertions to align with the output we receive back from Genie. It’s also valuable to show the possibility of having a single playbook using any parser to run operational assertions. If we were in production, we could make the <code class="highlighter-rouge">convert_data</code> custom filter plugin translate several different parser formats into a parser agnostic format.</p> <p>For brevity, here is our <code class="highlighter-rouge">tree</code> output to show you what the folder structure looks like to use a custom filter plugin.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ tree <span class="nb">.</span> ├── ansible.cfg ├── filter_plugins │   └── custom_filters.py ├── group_vars │   ├── all │   │   └── all.yml │   └── ios.yml ├── host_vars │   ├── iosv-0.yml │   ├── iosv-1.yml │   └── iosv-2.yml ├── inventory └── pb.validate.neighbors.yml </code></pre></div></div> <p>Finally the contents of the <code class="highlighter-rouge">custom_filters.py</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># -*- coding: utf-8 -*- </span> <span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">absolute_import</span><span class="p">,</span> <span class="n">division</span><span class="p">,</span> <span class="n">print_function</span> <span class="n">__metaclass__</span> <span class="o">=</span> <span class="nb">type</span> <span class="kn">import</span> <span class="nn">re</span> <span class="k">def</span> <span class="nf">convert_genie_data</span><span class="p">(</span><span class="n">data</span><span class="p">):</span> <span class="n">intfs</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span><span class="n">v</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s">'interfaces'</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="n">intf_name</span> <span class="o">=</span> <span class="s">"Gi"</span> <span class="o">+</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s">r'\d/\d'</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">intf_dict</span> <span class="o">=</span> <span class="p">{}</span> <span class="n">intf_dict</span><span class="p">[</span><span class="s">'local_interface'</span><span class="p">]</span> <span class="o">=</span> <span class="n">intf_name</span> <span class="n">neighbor_intf</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">v</span><span class="p">[</span><span class="s">'port_id'</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span> <span class="n">intf_dict</span><span class="p">[</span><span class="s">'neighbor'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">v</span><span class="p">[</span><span class="s">'port_id'</span><span class="p">][</span><span class="n">neighbor_intf</span><span class="p">][</span><span class="s">'neighbors'</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span> <span class="n">intfs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">intf_dict</span><span class="p">)</span> <span class="k">return</span> <span class="n">intfs</span> <span class="k">class</span> <span class="nc">FilterModule</span><span class="p">:</span> <span class="k">def</span> <span class="nf">filters</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="n">filters</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'convert_data'</span><span class="p">:</span> <span class="n">convert_genie_data</span><span class="p">,</span> <span class="p">}</span> <span class="k">return</span> <span class="n">filters</span> </code></pre></div></div> <p>I hope you enjoyed this blog post and understand a little bit more about Genie parsers and how to consume them with Ansible. The next post in this series will go over Ansible Engine parsing.</p> <p>-Mikhail</p>Mikhail YohmanThank you for joining me for Part 3 of the parsing strategies blog series. This post will dive deeper into using Cisco’s PyATS Genie library for parsing. For further reference in this blog post, I’ll be referring to the Genie library just by Genie. Genie also uses Regular Expressions (RegEx) under the hood to parse the output received from a device whether it’s semi-structured data, XML, YANG, etc. This is a key difference from other parsers, but for now, let’s stick with parsing semi-structured data.Getting Started Using the Kubernetes Collection in Ansible Tower2020-11-17T00:00:00+00:002020-11-17T00:00:00+00:00https://blog.networktocode.com/post/kubernetes-collection-ansible<p>In this post I’ll cover setting up and working with the <a href="https://github.com/ansible-collections/community.kubernetes">community.kubernetes</a> collection in Ansible Tower. I’ll describe my experience with the initial installation of the collection on the Ansible Tower controller, discuss using a Custom Credential Type in Ansible Tower for authentication to the Kubernetes API, and cover retrieving and parsing the output from the Kubernetes cluster. Finally, I’ll provide a sample playbook to create a pod in Kubernetes and query the pod status using the collection.</p> <p>While the main topic and examples are focused on using the <code class="highlighter-rouge">community.kubernetes</code> collection, much of the information here is applicable to other Ansible collections or modules as well. For example, getting Tower to recognize a collection, creating and using a Custom Credential Type, and parsing output using the <code class="highlighter-rouge">json_query</code> filter are very relevant when using any Ansible module or collection.</p> <h2 id="overview">Overview</h2> <p>Ansible collections were introduced recently as a way to help scale Ansible development by allowing the maintainers of the modules to manage them outside of Ansible core. As part of this transition, the core Kubernetes modules have been migrated to the <a href="https://github.com/ansible-collections/community.kubernetes">community.kubernetes</a> collection. The modules have the same functionality and syntax as the previous core modules, but will be maintained in the collection going forward. With that being the case, it is recommended to begin utilizing the collection rather than the core modules. The goal of this post is to help the reader get started with using the collection, specifically on Ansible Tower or AWX.</p> <h2 id="setup">Setup</h2> <p>First, it is necessary to install the <code class="highlighter-rouge">openshift</code> Python module which is used by the <code class="highlighter-rouge">community.kubernetes</code> modules on the Ansible Tower controller. By default Ansible Tower uses a virtual environment located at <code class="highlighter-rouge">/var/lib/awx/venv/</code>, so you must first activate that virtual environment and then use <code class="highlighter-rouge">pip</code> to install the module.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>centos@ip-10-125-1-252 ~]<span class="nv">$ </span><span class="nb">source</span> /var/lib/awx/venv/ansible/bin/activate <span class="o">(</span>ansible<span class="o">)</span> <span class="o">[</span>centos@ip-10-125-1-252 ~]<span class="nv">$ </span>pip <span class="nb">install </span>openshift </code></pre></div></div> <blockquote> <p>The above requires elevated privileges. If you are not running as root but have sudo privileges, run the command by specifying the full path to the pip utility: <code class="highlighter-rouge">sudo /var/lib/awx/venv/ansible/bin/pip install openshift</code></p> </blockquote> <p>The next step is installing the collection on the Ansible Tower controller and ensuring Tower is able to find the modules in the collection. I have been able to verify two methods that work on a Tower controller running Ansible 2.9.7 and Tower 3.6.3:</p> <h3 id="option-1-downloading-the-collection-on-demand"><strong>Option 1: Downloading the Collection On-Demand</strong></h3> <p>With this option, Ansible Tower will download the collection each time the playbook is run. Create a folder in the root of your project called <code class="highlighter-rouge">collections</code> and inside that folder include a <code class="highlighter-rouge">requirements.yml</code> file with contents such as below:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">collections</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">community.kubernetes</span> <span class="na">version</span><span class="pi">:</span> <span class="s">0.9.0</span> </code></pre></div></div> <p>The directory structure of the project would thus be:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── collections └── requirements.yml </code></pre></div></div> <p>This might not be desirable due to the delay incurred when downloading the collection, in which case you can opt for Option #2 below.</p> <blockquote> <p>Caching of collections may remediate this in future versions of Tower/AWX. See issue <a href="&quot;https://github.com/ansible/awx/pull/7643">#7643</a>. Thanks to NTC vet Josh VeDeraa for the tip!</p> </blockquote> <h3 id="option-2-include-a-copy-of-the-collection-in-your-source-control-repository"><strong>Option 2: Include a copy of the collection in your source control repository</strong></h3> <p>In order to avoid having to pull the collection on each playbook run, the collection can be installed as a folder in your project. This means it will be checked into your source control repository and be tracked along with your playbooks and other files. To accomplish this, first we need to create an <code class="highlighter-rouge">ansible.cfg</code> file in the root of the repository and create the following lines:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[defaults] collections_paths = ./ </code></pre></div></div> <p>Then run the <code class="highlighter-rouge">ansible-galaxy</code> command to install a copy of the collection into an <code class="highlighter-rouge">ansible_collections</code> folder inside your project. The requirements file should have the same contents as shown above in Option #1.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-galaxy collection install -r requirements.yml </code></pre></div></div> <p>This will create an <code class="highlighter-rouge">ansible_collections</code> folder at the root of your repo which contains the Kubernetes collection. The directory structure in the project will then be:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── ansible.cfg ├── ansible_collections │ └── community │ └── kubernetes └── requirements.yml </code></pre></div></div> <p>Commit the newly created <code class="highlighter-rouge">ansible_collections</code> folder into the repo, and make sure to re-sync the Project in Ansible Tower.</p> <h3 id="avoiding-the-unresolved-module-problem-on-the-tower-controller">Avoiding the “unresolved module” problem on the Tower controller</h3> <p>When I originally began working on this project, I began by installing the collection locally on the Tower controller using Ansible Galaxy which would typically be the way to install collections. As it turned out, installing the module locally on the controller and then trying to get Ansible Tower to find the collections is difficult or may even be impossible, at least on our version of Tower (3.6.3). For example, you might be tempted to do this on the controller:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-galaxy collection install community.kubernetes </code></pre></div></div> <p>This pulls down the collection to the Tower controller, however if you attempt to launch a template using one of the collection modules you will get an error such as:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR! couldn't resolve module/action 'community.kubernetes.k8s_info'. This often indicates a misspelling, missing collection, or incorrect module path. </code></pre></div></div> <p>Even after locating where Galaxy installed the collections, <code class="highlighter-rouge">/home/centos/.ansible/collections</code> on our Tower system, and then putting this path under the <code class="highlighter-rouge">collections_paths</code> key in <code class="highlighter-rouge">/etc/ansible/ansible.cfg</code>, the modules still could not be located.</p> <p>It seems like this should have worked, but at least on our version of Tower it did not. The options in the previous section were discovered only after many hours of trying to make this work, so save yourself some time!</p> <h3 id="testing-that-tower-can-find-the-module"><strong>Testing that Tower can find the module</strong></h3> <p>Now that we have set things up to allow Tower to successfully find the collection, we can test it with a playbook that uses the collection. The below playbook uses the <code class="highlighter-rouge">k8s_info</code> module to gather information about the Kubernetes worker nodes.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">RETRIEVE WORKER NODE DETAILS FROM KUBERNETES NODE</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">collections</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">community.kubernetes</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GET WORKER NODE DETAILS</span> <span class="s">community.kubernetes.k8s_info</span><span class="pi">:</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Node</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> </code></pre></div></div> <p>The playbook called <code class="highlighter-rouge">pb_test_setup.yml</code> is checked into our GitHub repository, which is set up as a Project in Ansible Tower. It is then referenced in the below Template:</p> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/01-tower-template.png" alt="Template" /></p> <p>If we run this playbook from the Tower controller, it will fail because we haven’t yet defined our credentials to access the control plane. A large amount of output will be displayed, but if you scroll up to the beginning of the output for the task <code class="highlighter-rouge">GET WORKER NODE DETAILS</code>, it should show the following output:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>An exception occurred during task execution. To see the full traceback, use -vvv. The error was: kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found. </code></pre></div></div> <p>This means that the module has been found, but we cannot connect to the Kubernetes control plane yet because we haven’t defined a kube-config file which provides the credentials to do so. In the next section I’ll show how to do that by defining a Custom Credential in Ansible Tower.</p> <h2 id="handling-credentials">Handling Credentials</h2> <p>To access the Kubernetes control plane, the credentials are typically stored in a <code class="highlighter-rouge">Kubeconfig</code> file which is stored locally on the user workstation in the file <code class="highlighter-rouge">~/.kube/config</code> by default. To securely store the credentials on the Ansible Tower controller, we can load the contents of the Kubeconfig file into a Credential in Ansible Tower. Tower does not currently have a native Kubernetes credential type, but it does provide the ability to create custom Credential Types. The steps below show how to create a Custom Credential Type in Ansible Tower to support storing of the Kubeconfig contents as a Credential. First a Kubernetes Credential Type is created, and then a Credential is created using the new credential type which stores the contents of the Kubeconfig file. We’ll then apply the Credential to the Job Template.</p> <ol> <li> <p>Navigate to Administration &gt; Credential Types and add a new credential type.</p> </li> <li> <p>Enter the following YAML in the Input Configuration:</p> </li> </ol> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">fields</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">id</span><span class="pi">:</span> <span class="s">kube_config</span> <span class="na">type</span><span class="pi">:</span> <span class="s">string</span> <span class="na">label</span><span class="pi">:</span> <span class="s">kubeconfig</span> <span class="na">secret</span><span class="pi">:</span> <span class="no">true</span> <span class="na">multiline</span><span class="pi">:</span> <span class="no">true</span> <span class="na">required</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">kube_config</span> </code></pre></div></div> <p>This defines a single input field in the credential that will be used to store the contents of the Kubeconfig file. Setting <code class="highlighter-rouge">secret</code> to true will cause the contents to be hidden in the Tower UI once saved. The <code class="highlighter-rouge">label</code> defines a caption for the field, and setting <code class="highlighter-rouge">multiline</code> to true causes a text entry field to be created which accepts multiple lines of text.</p> <ol> <li>Enter the below YAML in the Injector Configuration: <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">env</span><span class="pi">:</span> <span class="na">K8S_AUTH_KUBECONFIG</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">tower.filename.kubeconfig</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">file</span><span class="pi">:</span> <span class="s">template.kubeconfig</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">kube_config</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div> </div> <p>The Injector Configuration provides the ability to store the credential data input by the user on the Tower controller system in a way that allows it to be later referenced by an Ansible playbook, such as environment or extra variables. The above Injector Configuration will create a file called <code class="highlighter-rouge">kubeconfig</code> on the controller containing the content the user enters in the <code class="highlighter-rouge">kube_config</code> field of the Credential. The contents of the <code class="highlighter-rouge">kubeconfig</code> file created on the controller are then stored inside an environment variable on the Tower controller. The modules in the <code class="highlighter-rouge">community.kubernetes</code> collection by default look for the Kubeconfig in the environment variable <code class="highlighter-rouge">K8S_AUTH_KUBECONFIG</code>.</p> </li> </ol> <blockquote> <p>Injector Configuration is described <a href="https://docs.ansible.com/ansible-tower/latest/html/userguide/credential_types.html">here</a> in the Ansible documentation</p> </blockquote> <p>When complete the configuration should look similar to the image below:</p> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/02-custom-credential-type.png" alt="CustomCred" /></p> <ol> <li> <p>Navigate to Credentials and add a new Credential.</p> </li> <li> <p>In the Credential Type, select the Kubernetes custom Credential Type that was just created. In the Kubeconfig field, copy/paste the contents of your Kubeconfig file (usually <code class="highlighter-rouge">~/.kube/config</code>). Your configuration should look similar to the following:</p> </li> </ol> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/03-kubernetes-credential.png" alt="Credential" /></p> <blockquote> <p>Once you click Save, notice that the Kubeconfig contents are encrypted</p> </blockquote> <ol> <li>Apply the configuration to the Job Template. Navigate to Resources &gt; Templates and under Credentials search for the Kubernetes credential we just defined. Under Credential Type select Kubernetes.</li> </ol> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/04-select-credential-type.png" alt="SelectCredential" /></p> <p>When finished, your template should look similar to this:</p> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/05-template-with-credential.png" alt="TemplateWithCred" /></p> <p>We are now ready to execute the playbook in Tower. If all is working correctly, we should have a successful job completion…</p> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/06-get-info-job-success.png" alt="GetInfoJobSuccess" /></p> <blockquote> <p>In the case of our setup we were using Amazon EKS for our Kubernetes cluster, so we also had to install the AWS CLI on the Tower controller, create an AWS credential in Tower containing the AWS API keys, and apply the AWS credential to the job template.</p> </blockquote> <p>As you can see, the job was successful but it wasn’t very exciting because we aren’t yet displaying any of the information that we collected. In the next section, we’ll look at how to display and parse the output.</p> <h2 id="displaying-and-parsing-output">Displaying and Parsing Output</h2> <p>In order to display data, we first need to register a variable to store the output of the task and then create a second task using the debug module to display the variable. In the below playbook, I have added the <code class="highlighter-rouge">register</code> command to the initial task to store the output of the task in a variable called <code class="highlighter-rouge">node_result</code>, and added a debug task below it to display the registered variable.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GET WORKER NODE DETAILS</span> <span class="s">community.kubernetes.k8s_info</span><span class="pi">:</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Node</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> <span class="na">register</span><span class="pi">:</span> <span class="s">node_result</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DISPLAY OUTPUT</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">node_result</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>When the above playbook is run, the debug task will display a large amount of data about the Kubernetes worker node(s).</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"msg"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"changed"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"resources"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-13-155.us-east-2.compute.internal"</span><span class="p">,</span><span class="w"> </span><span class="nl">"selfLink"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/api/v1/nodes/ip-172-17-31-155.us-east-2.compute.internal"</span><span class="p">,</span><span class="w"> </span><span class="nl">"uid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"27a57359-4019-458e-988f-9d4984e74662"</span><span class="p">,</span><span class="w"> </span><span class="nl">"resourceVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"275328"</span><span class="p">,</span><span class="w"> </span><span class="nl">"creationTimestamp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-29T18:52:11Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"labels"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"beta.kubernetes.io/arch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"amd64"</span><span class="p">,</span><span class="w"> </span><span class="nl">"beta.kubernetes.io/instance-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"t3.medium"</span><span class="p">,</span><span class="w"> </span><span class="nl">"beta.kubernetes.io/os"</span><span class="p">:</span><span class="w"> </span><span class="s2">"linux"</span><span class="p">,</span><span class="w"> </span><span class="nl">"eks.amazonaws.com/nodegroup"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eks-nodegroup"</span><span class="p">,</span><span class="w"> </span><span class="nl">"eks.amazonaws.com/nodegroup-image"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ami-0c619f57dc7e552a0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"eks.amazonaws.com/sourceLaunchTemplateId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lt-0ac1bb9bae7ecb7f6"</span><span class="p">,</span><span class="w"> </span><span class="nl">"eks.amazonaws.com/sourceLaunchTemplateVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"failure-domain.beta.kubernetes.io/region"</span><span class="p">:</span><span class="w"> </span><span class="s2">"us-east-2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"failure-domain.beta.kubernetes.io/zone"</span><span class="p">:</span><span class="w"> </span><span class="s2">"us-east-2a"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kubernetes.io/arch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"amd64"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kubernetes.io/hostname"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-13-155.us-east-2.compute.internal"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kubernetes.io/os"</span><span class="p">:</span><span class="w"> </span><span class="s2">"linux"</span><span class="p">,</span><span class="w"> </span><span class="nl">"node.kubernetes.io/instance-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"t3.medium"</span><span class="p">,</span><span class="w"> </span><span class="nl">"topology.kubernetes.io/region"</span><span class="p">:</span><span class="w"> </span><span class="s2">"us-east-2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"topology.kubernetes.io/zone"</span><span class="p">:</span><span class="w"> </span><span class="s2">"us-east-2a"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"annotations"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"node.alpha.kubernetes.io/ttl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"volumes.kubernetes.io/controller-managed-attach-detach"</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"spec"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"providerID"</span><span class="p">:</span><span class="w"> </span><span class="s2">"aws:///us-east-2a/i-0ba92f585eebfae76"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"capacity"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"attachable-volumes-aws-ebs"</span><span class="p">:</span><span class="w"> </span><span class="s2">"25"</span><span class="p">,</span><span class="w"> </span><span class="nl">"cpu"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ephemeral-storage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"20959212Ki"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hugepages-1Gi"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hugepages-2Mi"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"memory"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3977908Ki"</span><span class="p">,</span><span class="w"> </span><span class="nl">"pods"</span><span class="p">:</span><span class="w"> </span><span class="s2">"17"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"allocatable"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"attachable-volumes-aws-ebs"</span><span class="p">:</span><span class="w"> </span><span class="s2">"25"</span><span class="p">,</span><span class="w"> </span><span class="nl">"cpu"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1930m"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ephemeral-storage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"18242267924"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hugepages-1Gi"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hugepages-2Mi"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"memory"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3422900Ki"</span><span class="p">,</span><span class="w"> </span><span class="nl">"pods"</span><span class="p">:</span><span class="w"> </span><span class="s2">"17"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"conditions"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"MemoryPressure"</span><span class="p">,</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"False"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastHeartbeatTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-30T20:24:06Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastTransitionTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-29T18:52:11Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"reason"</span><span class="p">:</span><span class="w"> </span><span class="s2">"KubeletHasSufficientMemory"</span><span class="p">,</span><span class="w"> </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubelet has sufficient memory available"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DiskPressure"</span><span class="p">,</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"False"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastHeartbeatTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-30T20:24:06Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastTransitionTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-29T18:52:11Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"reason"</span><span class="p">:</span><span class="w"> </span><span class="s2">"KubeletHasNoDiskPressure"</span><span class="p">,</span><span class="w"> </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubelet has no disk pressure"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"PIDPressure"</span><span class="p">,</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"False"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastHeartbeatTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-30T20:24:06Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastTransitionTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-29T18:52:11Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"reason"</span><span class="p">:</span><span class="w"> </span><span class="s2">"KubeletHasSufficientPID"</span><span class="p">,</span><span class="w"> </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubelet has sufficient PID available"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ready"</span><span class="p">,</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"True"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastHeartbeatTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-30T20:24:06Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lastTransitionTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-09-29T18:52:31Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"reason"</span><span class="p">:</span><span class="w"> </span><span class="s2">"KubeletReady"</span><span class="p">,</span><span class="w"> </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubelet is posting ready status"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"addresses"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"InternalIP"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"172.17.31.155"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ExternalIP"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"13.14.131.143"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Hostname"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-31-155.us-east-2.compute.internal"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"InternalDNS"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-31-155.us-east-2.compute.internal"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ExternalDNS"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ec2-13-14-131-143.us-east-2.compute.amazonaws.com"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"daemonEndpoints"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"kubeletEndpoint"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"Port"</span><span class="p">:</span><span class="w"> </span><span class="mi">10250</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"nodeInfo"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"machineID"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ec2475ce297619b1bcfe0d56602b5284"</span><span class="p">,</span><span class="w"> </span><span class="nl">"systemUUID"</span><span class="p">:</span><span class="w"> </span><span class="s2">"EC2475CE-2976-19B1-BCFE-0D56602B5284"</span><span class="p">,</span><span class="w"> </span><span class="nl">"bootID"</span><span class="p">:</span><span class="w"> </span><span class="s2">"d591d21a-46de-4011-8941-12d72cb5950e"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kernelVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4.14.193-149.317.amzn2.x86_64"</span><span class="p">,</span><span class="w"> </span><span class="nl">"osImage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Amazon Linux 2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"containerRuntimeVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"docker://19.3.6"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kubeletVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"v1.17.11-eks-cfdc40"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kubeProxyVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"v1.17.11-eks-cfdc40"</span><span class="p">,</span><span class="w"> </span><span class="nl">"operatingSystem"</span><span class="p">:</span><span class="w"> </span><span class="s2">"linux"</span><span class="p">,</span><span class="w"> </span><span class="nl">"architecture"</span><span class="p">:</span><span class="w"> </span><span class="s2">"amd64"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"images"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"ceosimage/4.24.2.1f:latest"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">1768555566</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni@sha256:400ab98e321d88d57b9ffd15df51398e6c2c6c0167a25838c3e6d9637f6f5e0c"</span><span class="p">,</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni:v1.6.3-eksbuild.1"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">282945379</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"nfvpe/multus@sha256:9a43e0586a5e6cb33f09a79794d531ee2a6b97181cae12a82fcd2f2cd24ee65a"</span><span class="p">,</span><span class="w"> </span><span class="s2">"nfvpe/multus:stable"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">277329369</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy@sha256:cbb2c85cbaa3d29d244eaec6ec5a8bbf765cc651590078ae30e9d210bac0c92a"</span><span class="p">,</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">130676901</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns@sha256:476c154960a843ac498376556fe5c42baad2f3ac690806b9989862064ab547c2"</span><span class="p">,</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.6.6-eksbuild.1"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">40859174</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause@sha256:1cb4ab85a3480446f9243178395e6bee7350f0d71296daeb6a9fdd221e23aea6"</span><span class="p">,</span><span class="w"> </span><span class="s2">"602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause:3.1-eksbuild.1"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"sizeBytes"</span><span class="p">:</span><span class="w"> </span><span class="mi">682696</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"apiVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"v1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Node"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"ansible_facts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"discovered_interpreter_python"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/usr/libexec/platform-python"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"failed"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"_ansible_verbose_always"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nl">"_ansible_no_log"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"changed"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>What if we are only looking for a subset of that data? This is where a handy filter that is available in Ansible called <code class="highlighter-rouge">json_query</code> comes in. The <code class="highlighter-rouge">json_query</code> filter can be invoked by using the pipe <code class="highlighter-rouge">|</code> symbol followed by the <code class="highlighter-rouge">json_query</code> command and passing in a query as an argument. A query can be built to parse the JSON output and return specific data that we are looking for. Let’s say we wanted to return only the IP addressing assigned to the worker node. To do this, let’s add another set of tasks to the playbook:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SET VARIABLE STORING EKS WORKER NODE ADDRESS</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">eks_worker_address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">node_result</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">json_query(query)</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">query</span><span class="pi">:</span> <span class="s2">"</span><span class="s">resources[].status.addresses"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DISPLAY EKS WORKER ADDRESS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">eks_worker_address</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>Here we are using the <code class="highlighter-rouge">set_fact</code> module to set another variable called <code class="highlighter-rouge">eks_worker_address</code> that will contain our filtered results. The task variable <code class="highlighter-rouge">query</code> contains the query syntax that is passed to the <code class="highlighter-rouge">json_query</code> filter above. Breaking down the query, in the previous <code class="highlighter-rouge">node_result</code> variable output it can be seen that <code class="highlighter-rouge">resources</code> is a List containing many elements. The double brackets <code class="highlighter-rouge">[]</code> next to resources in the query instructs the filter to return all elements in the list. Next, under resources we have a number of dictionary keys, one of which is the <code class="highlighter-rouge">status</code> key. The <code class="highlighter-rouge">.status</code> after <code class="highlighter-rouge">resources[]</code> will filter the results to only content under the <code class="highlighter-rouge">status</code> key. Finally we have <code class="highlighter-rouge">.addresses</code>, which is a key that references another list containing all addresses on the worker node. When the task is run, the output should look similar to this:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"msg"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"InternalIP"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"172.17.13.155"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ExternalIP"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3.14.130.143"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Hostname"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-13-155.us-east-2.compute.internal"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"InternalDNS"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ip-172-17-13-155.us-east-2.compute.internal"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ExternalDNS"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ec2-3-14-130-143.us-east-2.compute.amazonaws.com"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">]</span><span class="w"> </span></code></pre></div></div> <p>What if we wanted to filter this further and just return the External DNS name of our Kubernetes worker node? To accomplish this, we can filter on the <code class="highlighter-rouge">type</code> key in the list of addresses for the “ExternalDNS” value. To do this, the query in the task can be modified as shown below:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SET VARIABLE STORING EKS WORKER NODE ADDRESS</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">eks_worker_address</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">node_result</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">json_query(query)</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">query</span><span class="pi">:</span> <span class="s2">"</span><span class="s">resources[].status.addresses[?type=='ExternalDNS']"</span> </code></pre></div></div> <p>Now the output is limited to just the ExternalDNS…</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"msg"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ExternalDNS"</span><span class="p">,</span><span class="w"> </span><span class="nl">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ec2-3-14-130-143.us-east-2.compute.amazonaws.com"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"_ansible_verbose_always"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nl">"_ansible_no_log"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"changed"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>This only scratches the surface of what is possible when parsing with the <code class="highlighter-rouge">json_query</code> filter. Behind the scenes, Ansible is using a utility called JMESPATH. The query language in JMESPATH is the same syntax that is used for the query argument passed to the <code class="highlighter-rouge">json_query</code> filter. Have a look at the <a href="https://jmespath.org/tutorial.html">documentation</a> to learn more.</p> <h2 id="deploying-kubernetes-resources">Deploying Kubernetes Resources</h2> <p>In this final section, I’ll discuss using the <code class="highlighter-rouge">community.kubernetes</code> <code class="highlighter-rouge">k8s</code> module to initiate deployment of a Busybox pod in Kubernetes. Busybox is a lightweight utility container that is often used to validate and troubleshoot deployments. The <code class="highlighter-rouge">k8s</code> module can be used to deploy resources into Kubernetes by launching them from a Kubernetes definition file. For those familiar with using the <code class="highlighter-rouge">kubectl</code> command line utility, this is equivalent to running the command <code class="highlighter-rouge">kubectl apply -f [deployment_file]</code>. Given that we will be deploying a pod from a definition file, we must first create the definition file:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">busybox</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">busybox</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">busybox</span> <span class="na">image</span><span class="pi">:</span> <span class="s">busybox</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">sleep'</span><span class="pi">,</span><span class="s1">'</span><span class="s">3600'</span><span class="pi">]</span> <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span> <span class="na">restartPolicy</span><span class="pi">:</span> <span class="s">Always</span> </code></pre></div></div> <p>The above yaml should be saved as <code class="highlighter-rouge">busybox.yml</code>. Next we create an Ansible playbook called <code class="highlighter-rouge">pb_deploy_busybox.yml</code> that will use the <code class="highlighter-rouge">k8s</code> module to apply the definition file.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DEPLOY BUSYBOX</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DEPLOY BUSYBOX TO KUBERNETES</span> <span class="s">community.kubernetes.k8s</span><span class="pi">:</span> <span class="na">definition</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lookup('template',</span><span class="nv"> </span><span class="s">'busybox.yml')</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">from_yaml</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">state</span><span class="pi">:</span> <span class="s">present</span> </code></pre></div></div> <p>Commit this to the the source control repository, and synchronize the project in Ansible Tower. The Job Template in Ansible Tower should look similar to this:</p> <p><img src="../../../static/images/blog_posts/kubernetes-collection-ansible/07-busybox-template.png" alt="BusyboxTemplate" /></p> <p>Here’s the output that will be shown when the template is launched in Ansible Tower:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PLAY [DEPLOY BUSYBOX] ********************************************************** 09:44:35 TASK [DEPLOY BUSYBOX TO KUBERNETES] ******************************************** 09:44:35 changed: [localhost] PLAY RECAP ********************************************************************* 09:44:37 localhost : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre></div></div> <p>From the above output we can assume the pod was deployed successfully, and we could confirm from the command line using the <code class="highlighter-rouge">kubectl get pod</code> command. However, we can also use a similar output parsing strategy as was used in the previous section to gather information about running pods. Here we’ll use the <code class="highlighter-rouge">k8s_info</code> module to report on running Kubernetes pods, and filter the output using the <code class="highlighter-rouge">json_query</code> filter. Add another playbook called <code class="highlighter-rouge">pb_display_pod_info.yml</code>.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">RETRIEVE POD DETAILS FROM KUBERNETES</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">collections</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">community.kubernetes</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GET POD DETAILS</span> <span class="s">community.kubernetes.k8s_info</span><span class="pi">:</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Pod</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> <span class="na">register</span><span class="pi">:</span> <span class="s">pod_result</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DISPLAY OUTPUT</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">pod_result</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>This once again yields a lot of data, on which we can use <code class="highlighter-rouge">json_query</code> to filter the output to what we want to see. Let’s add a few more tasks to filter the output.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FILTER FOR POD NAME AND STATE</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">filtered_result</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">pod_result</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">json_query(query)</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">query</span><span class="pi">:</span> <span class="s2">"</span><span class="s">resources[].{name:</span><span class="nv"> </span><span class="s">status.containerStatuses[].name,</span><span class="nv"> </span><span class="s">status:</span><span class="nv"> </span><span class="s">status.containerStatuses[].state}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">DISPLAY FILTERED RESULTS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">filtered_result</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>This time we are using a slightly different syntax for the query which creates a dictionary containing the values selected from the larger JSON output produced in the previous task, and that is stored in <code class="highlighter-rouge">pod_result</code>. Within the curly brackets <code class="highlighter-rouge">{}</code>, the <code class="highlighter-rouge">name</code> key is what we chose to hold the value retrieved by referencing status.containerStatuses[].name. Similarly, the <code class="highlighter-rouge">status</code> key holds the state of the container retrieved from status.containerStatuses[].state. The final output should be similar to below.</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"msg"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"busybox"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"running"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"startedAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2020-10-01T13:44:38Z"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span></code></pre></div></div> <p>This method of filtering and storing data can be used not only for display purposes, but also to dynamically populate variables that can later be used in sub-sequent Ansible tasks if needed.</p> <p>That does it for this post, I hope it provides some useful hints to get up and running quickly automating Kubernetes with Ansible Tower!</p> <p>-Matt</p>Matt MullenIn this post I’ll cover setting up and working with the community.kubernetes collection in Ansible Tower. I’ll describe my experience with the initial installation of the collection on the Ansible Tower controller, discuss using a Custom Credential Type in Ansible Tower for authentication to the Kubernetes API, and cover retrieving and parsing the output from the Kubernetes cluster. Finally, I’ll provide a sample playbook to create a pod in Kubernetes and query the pod status using the collection.