Jekyll2024-02-16T15:03:39+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comNetwork Management Fundamentals - Getting Started with SoT & Nautobot2024-02-16T00:00:00+00:002024-02-16T00:00:00+00:00https://blog.networktocode.com/post/network-management-fundamentals-getting-started-with-sot-and-nautobotLooking to learn how you can get started with Nautobot? You’ve come to the right place.

During our monthly webinars in 2023, Tim Schreyack, Director of Sales Engineering at Network to Code, and Jeremy White, Principal Developer Advocate at Network to Code, came together to discuss why data is the key to unlocking successful enterprise network automation.

Their conversations covered the current state of network management, why an authoritative Network Source of Truth is necessary for a network automation framework, the importance of a validated and well-documented network intended state, and how Nautobot helps organizations take back control of their network with a data-driven approach to network automation.

Read on to learn more about how you can eliminate spreadsheets and efficiently manage everything from inventory and IPAM to BGP, firewall policies, and much more with Nautobot!

The State of Network Management

In the current state of network management, most organizations are leveraging tools, processes, and technologies developed over 20 years ago.

As companies grow, so do the scale and complexity of their networks. Eventually, this growth causes all manual CLI-driven methodologies of network management to become unviable. Commonly, what we see happening in organizations is a transition from all manual CLI to personal scripts, whether batch shell or Python scripts, that are deployed on individual computers to help maintain the network environment. Some organizations take the next step and collaborate on these power tools.

Despite the popularity of this approach to network management, the legacy management of the network ultimately succumbs to its numerous costs and inefficiencies. Not only does this approach leave engineers spending the majority of their time on monotonous tasks, but the operating costs of maintaining the environment are extremely high. Legacy management of the network also results in slow response times and excessive backlogs, limiting scalability.

As organizations continue down this journey of trying to evolve their legacy practices for network management, they discover the need for a Network Source of Truth (NSoT). This is because as the maturity of enterprise automation frameworks increases, an NSoT is necessary to increase the efficiency of network operations as well.

Companies cannot drive automation and change without an NSoT.

Introduction to NSoT

Change in network management begins with data.

“No enterprise should develop network automations on top of systemic uncertainty of the underlying in-service network data. Unfortunately, 74% of a typical enterprise’s network inventory data is partially, minimally, or not at all accurate. Network automations require a reliable NSoT,” says Gartner.

If all of an organization’s data is accessible from a Network Source of Truth (NSoT), there is the opportunity to not only drive network automation but also perform richer capacity planning, trending, and business impact analysis.

The key to understanding NSoT is that it is all about intentions and planning. Once we express our intent into the NSoT, that becomes the authoritative state for the environment against which as-is network behaviors are compared. The difference between the intended state and the as-is state identifies anomalies for investigation and correction, with the corrective action being centered on simply implementing any needed changes to the data. This is why Network to Code refers to our approach as data-driven network automation.

The more accurate the data is, the more insights we will gain.

The Power of Nautobot

For those that are unfamiliar with it, Nautobot is an extensible and flexible Network Source of Truth and Network Automation Platform that can power any network automation architecture. It is not just about the Source of Truth. It’s about how we use that Source of Truth to drive change and integrate tools into the environment.

Network to Code created Nautobot as an open source community project, purpose-built to automate network management.

One of the challenges we face as we move into an automated world is that it is very easy to go from making a change that might only impact one device to making a change that might impact a lot of devices. With this expanded impact of automation, we must ensure our data is accurate and driving the change we expect it to, because automation is only as good as the data behind it.

Nautobot supports two main ways of enforcing data validation and integrity: the Data Validation Application and User-Defined Jobs. The Data Validation Application API provides a way to express business logic into the data. Data Validation Application is a no-code solution that allows administrators to define rule-based data validation during the creation and modification of objects. User-Defined Jobs can leverage the Data Compliance feature of the Data Validation Engine to verify compliance across preexisting objects.

Nautobot Key Features for Network Source of Truth

At its core, we’ve discussed how Nautobot is a Network Source of Truth that defines the intended state of the network, but let’s dive into its key network management features.

Data Model Features: With NSoT, you can codify business rules to ensure there is nothing but high-quality data in Nautobot. NSoT enables you to create custom relationships between existing data models that replicate your network design. On top of that, you can augment existing data models through custom fields on any object, including interfaces.

Highly Extensible: Free purpose-built applications cater to your specific NSoT and network automation requirements.

Highly Available Data: NSoT allows you to easily fetch the exact data you desire across data models with a single API call. Additionally, Nautobot can make an outbound HTTP API calls based on create, update, and delete operations.

All of these key features help to support data-driven network automation.

Getting Started with Nautobot and NSoT

Now that we have identified Nautobot’s key features and several ways to load and validate data about the network, what’s next for network management with Nautobot?

Here are a few recommendations to start leveraging Nautobot:

  1. Populate interface descriptions to describe what is connected and informed by the NSoT.
  2. Leverage ChatOps to retrieve information via your favorite chat application.
  3. Maintain a few simple configuration stanzas across your entire organization.
  4. Begin tracking device lifecycles programmatically.

Want to learn more about how aggregating your network model and data into Nautobot accelerates your network automation journey?

Check out this webinar replay to learn more and get your questions answered!

-Chris Murray

]]>
Chris Murray
Nautobot Cloud - Your Gateway to Network Automation2024-02-14T00:00:00+00:002024-02-14T00:00:00+00:00https://blog.networktocode.com/post/nautobot-cloud-your-gateway-to-network-automationNetwork to Code recently hosted a webinar on the future of network automation to demonstrate how you can elevate your network management experience with the leading SaaS network automation platform, Nautobot Cloud.

If your organization is ready to begin or improve your network automation journey with Nautobot, there are two deployment options to choose from: self-managed solutions (on-premises or within your established cloud assets) or cloud SaaS. Self-managed solutions refer to an approach where an organization’s IT infrastructure (whether physical or cloud-based) and resources are responsible for the deployment, operation, and maintenance of the enterprise software.

Embracing the future of network automation has never been easier since the introduction of Nautobot Cloud. Nautobot Cloud enables engineering teams to quickly operationalize a Network Source of Truth (NSoT) and network automation platform in a single open source SaaS platform.

Before we dive into the differences between a self-managed solution and Nautobot Cloud, it’s important to understand Nautobot and its core functionality.

What is Nautobot?

Nautobot is the most widely deployed open source Network SoT and network automation platform used by large enterprises today, offering a data-driven approach to network automation. It offers organizations superior flexibility, extensibility, and control while catering to any network design.

One of Nautobot’s greatest strengths is that data doesn’t have to reside within Nautobot; Nautobot offers a neutral model allowing the data to stay where it is most easily stored and accessed, but also ensuring all necessary data is identified and available. Nautobot enables bi-directional data flows so data can be enriched from each of the various sources, and combined as necessary to fulfill the data needs of the network automation tasks. With Nautobot as your NSoT, data about the network can be stored and exposed in a multitude of ways to support operations, including Rest APIs, JobHooks, GraphQL, Git Integration, and Webhooks. With Nautobot, even disparate data sources are unified, and compliance is more easily achieved, mitigating errors and security vulnerabilities.

Additionally, Nautobot maintains an intended network state that can be confidently documented, understood, and used as a baseline to compare current network behaviors against, empowering a more modern and data-driven NetDevOps approach to network configuration and management.

Nautobot Cloud

Exploring the Pros and Cons of Network Automation Deployment Options

There are several key pros and cons between a self-managed solution and a Cloud SaaS approach.

Deploying Nautobot through a self-managed approach is a more traditional method that many organizations are familiar with. However, these solutions come with their own challenges.

To start, a self-managed solution often requires major upfront costs for hardware (or cloud capacity), software, and setup, along with an ongoing need to pay for costly maintenance to maintain functionality and avoid obsolescence. Additionally this solution takes away time and effort that could otherwise be directed toward achieving business objectives, as team members must invest labor and hours towards learning to install, maintain, and use it.

On the other hand, Nautobot Cloud offers teams significant benefits when it comes to simplifying the adoption of a data-driven network automation platform.

First, while traditional self-managed solutions are notorious for lengthy installation times, Nautobot Cloud delivers an accelerated installation time. With Nautobot Cloud, installation is done in minutes, with a single click, saving your team countless hours and labor during the setup phase.

Nautobot Cloud is also an invaluable tool for optimizing your team. Thanks to the management of the infrastructure and Nautobot in the cloud, clients can access as many Nautobot instances as they need without investing time or training in any infrastructure development and maintenance.

Self-managed solutions also require multiple departments across the organization to function together, such as network automation teams, security teams, infrastructure teams, PMO teams, and network teams. With a cloud-based approach, a few dedicated team members handle NSoT population, configuration management, data access, automation jobs, and custom applications, leaving the rest of the organization to focus on their specialized roles and objectives.

Another major benefit of Nautobot Cloud is its comprehensive cloud native architecture features. With Nautobot Cloud, users gain access to:

  • Autoscaling of resources - easily scale and dynamically change your network maintenance, configuration, and security measures based on fluctuating traffic, data, or other needs
  • Secure multi-tenancy - segregate different lines of business, based on your specific requirements
  • Dedicated clusters - Nautobot Cloud uses a multi-cluster design, relieving worries about impact from parallel environment, unlike a traditional On-Prem solution
  • Highly Available - crucial for mission-critical applications, where system outages can have significant consequences.

By providing an easy-to-use interface to deploy scalable network automation, Nautobot Cloud enables teams to focus on boosting performance—not spending unnecessary time, effort, and budget on deployment complexities.

Nautobot Platform and Service Features

Now that we’ve discussed the benefits of choosing Cloud SaaS over an On-Prem solution for your network automation needs, let’s explore the key Nautobot features in a bit more detail.

Fast and easy installation and management

With Nautobot Cloud, cloud resources are created in minutes with a single click. Resource upgrades, cloning, and snapshots are also performed in one click. For organizations strapped on time and resources, Nautobot Cloud makes network automation and management faster, simpler, and more efficient.

Accelerated network automation

Nautobot is the key to accelerating your network automation adoption, thanks to its apps that solve common use cases. Each app is available in the marketplace and easily deployable via a 1-button integration model so you can quickly solve your most pressing challenges. Nautobot can also integrate with custom-made apps as needed.

Enterprise Grade Data Insights

Nautobot’s rich modern dashboard provides unique insights into your automation environment, including real-time dashboards that showcase data insights and trends in your network data and executed workflows.

Cloud Services Console

Another service feature of Nautobot Cloud is the ability for team members to access, use, and manage all cloud services through the web-based console. Cloud services include:

  • Create, clone, and upgrade Nautobot instances
  • Create and manage AWX deployments
  • Install Apps to Nautobot Instances
  • View dashboards

Nautobot Cloud Console

Full Automation Ecosystem

Nautobot Cloud is your full automation ecosystem, acting as a Network Source of Truth and your Network Automation platform. In addition to seamlessly working with Nautobot, Nautobot Cloud also enables efficient deployment and management of Ansible AWX. This means team members can truly install, manage, and maintain one of the most common network automation stacks, directly from Nautobot Cloud.

Learn More about Nautobot Cloud

Choosing between a self-managed solution and a cloud operational model is no longer a difficult choice with Nautobot Cloud. Nautobot Cloud is turbocharging the efficiency, productivity, and responsiveness of network teams across the globe.

Led by data-driven network automation, Nautobot Cloud is enabling engineering teams to quickly operationalize its Network Source of Truth and network automation platform in a single open source SaaS platform.

Ready to learn how Nabutobot Cloud can help streamline and improve your network automation journey?

Watch the full webinar here: https://www.youtube.com/watch?v=S8YhkV4oE1w

-Chris M.

]]>
Chris Murray
Last Month in Nautobot - January 20242024-02-09T00:00:00+00:002024-02-09T00:00:00+00:00https://blog.networktocode.com/post/last-month-in-nautobot-2401Welcome to our monthly Nautobot community update! We’ll dive into the latest updates across the Nautobot community highlighting the key milestones, releases, and noteworthy contributions. From new features and enhancements to bug fixes and events, there’s always something happening in our dynamic ecosystem. Open source is at the core of our values, empowering individuals and organizations to collaborate, innovate, and make a positive impact together. This monthly blog post is our way of celebrating the accomplishments and contributions of our Nautobot community members.

New Cookiecutter Templates Released for Nautobot App Development

We have released new Cookiecutters to help developers bootstrap their Nautobot App projects. Take a look at Jeremy’s blog post Introducing Cookiecutter Project Templates to Support Nautobot App Development for Network Automation to learn more!

Nautobot Core

Releases - Stable

  • Nautobot: v2.1.2 - 2024-01-22
    • #5054 - Added validation of redirect URLs to the “Add a new IP Address” and “Assign an IP Address” views
    • #5109 - Removed /files/get/ URL endpoint (for viewing FileAttachment files in the browser), as it was unused and could potentially pose security issues
    • #5133 - Fixed an XSS vulnerability (GHSA-v4xv-795h-rv4h) in the render_markdown() utility function used to render comments, notes, job log entries, etc.
    • #3877 - Added global filtering to Job Result log table, enabling search across all pages
    • Multiple bug fixes including some performance enhancements
  • Nautobot: v2.1.1 - 2024-01-08
    • #5046 - Updated the LocationType clone process to pre-populate the original object’s parent, nestable, and content type fields
    • #4992 - Added change-logging (ObjectChange support) for the ObjectPermission model
    • Multiple bug fixes for the GraphQL UI and saved queries

Releases - LTM 1.6

  • Nautobot: v1.6.10 - 2024-01-22
    • #5109 - Removed /files/get/ URL endpoint (for viewing FileAttachment files in the browser), as it was unused and could potentially pose security issues
    • #5134 - Fixed an XSS vulnerability (GHSA-v4xv-795h-rv4h) in the render_markdown() utility function used to render comments, notes, job log entries, etc.
    • #5134 - Enhanced Markdown-supporting fields (comments, description, Notes, Job log entries, etc.) to also permit the use of a limited subset of “safe” HTML tags and attributes
  • Nautobot: v1.6.9 - 2024-01-08
    • #5042 - Fixed early return conditional in ensure_git_repository

Apps Ecosystem

  • Nautobot App Device Lifecycle Mgmt: v1.6.0 - 2024-01-27
    • Nautobot LTM version 1.6 support
    • Add new tab “Contract devices” to the Contract details view
    • Bug fixes and performance enhancements
  • Nautobot App Device Lifecycle Mgmt: v2.1.0 - 2024-01-27
    • Bug fixes and performance enhancements
  • Nautobot Lab: 2.1.2 - 2024-01-26
    • Update to Nautobot v2.1.2
  • Nautobot Ansible: v5.1.1 - 2024-01-23
    • Remove status attribute requirement when state is present
  • Nautobot App SSoT: v2.2.0 - 2024-01-18
    • Add custom relationship capabilities to the contrib module
    • Update Infoblox SSoT to allow for gathering of IPv6 Prefixes
    • Bug fixes
  • Nautobot App Golden Config: v1.6.4 - 2024-01-11
    • Update Nautobot Nornir Dependency
  • Cookiecutter Nautobot App: v2.0.0 - 2024-01-09
    • Add templates for ChatOps Apps
    • Bug fixes for SSoT Apps
  • Nautobot App Firewall Models: v2.0.3 - 2024-01-08
    • Bug fixes
  • Nautobot App SSoT: v2.1.0 - 2024-01-05
    • Validate default settings exist for CVP integration and fix DeviceRole attribute
    • Add a debugging guide for SSoT jobs
    • Add Custom Database Parameter Loader
    • Fix bugs related to Nautobot v2.0 data model changes
    • Add single network view and IP Address type support
    • Add support for long interface names with IPFabric
    • Add support to IPFabric for additional interface media
  • Nautobot App Firewall Models: v2.0.2 - 2024-01-04
    • Documentation updates and bug fixes
  • Nornir Nautobot: v3.1.0 - 2024-01-04
    • Updates for Nautobot v2.0
    • Update Nornir logging
  • Cookiecutter Nautobot App: v1.2 - 2024-01-03
    • Nautobot App template improvements
    • Add Nautobot App SSoT template
  • Cookiecutter Nautobot App: v1.1 - 2024-01-03
    • Initial release of the Network to Code templates for Nautobot Apps

Community

Get in Touch!

Do you have any cool Nautobot-related project we should write about? Swing by the Network to Code Slack -> channel #nautobot and write us a quick line! Sign up here if you don’t have an account.

-Cristian and Gary

]]>
Cristian Sirbu and Gary Snider
Introducing Design Builder: Design Driven Network Automation2024-02-05T00:00:00+00:002024-02-05T00:00:00+00:00https://blog.networktocode.com/post/design-builderMost people involved in network automation are familiar with the concept of a Source of Truth (SoT). The SoT is usually some form of database that maintains intended state of objects as well as their interdependency. The SoT provides a way to quickly ascertain what a network’s intended state should be, while often providing a way to see what the network’s state actually is. A new concept is emerging, known as Design Oriented Source of Truth. This idea takes network designs and codifies them, attaching additional meaning to the objects within the SoT. Nautobot is a source of truth that contains all sorts of information about a network’s state. Although many of the pieces of information within Nautobot are related, they are discretely managed. A new Nautobot App aims to simplify the process of codifying network designs and populating Nautobot objects based on these designs.

Introduction

It is very common to have a small set of standardized designs that are used to deploy many sites and services in enterprise networks. For example, branch office sites may have a few different designs depending on their size. There could be a design that uses a single branch office router for small sites. Another design could have two routers and an access switch for sites with a moderate user base. A third design could include a more complex switching infrastructure for sites with many employees. When companies do tech refreshes or new site builds, these standardized designs are used and new data must be created in the source of truth. The newly open-sourced Design Builder application was created to address this problem, and fulfills the idea that a standardized design can be taken from a network engineer and transformed into a format that can be consumed and executed by Nautobot. Design Builder can expand a minimal set of inputs into a full-fledged set of configuration objects within Nautobot. This includes any kind of data object that Nautobot can model. Everything from Rack and Device objects to IP addresses and BGP peering information.

Design Builder provides powerful mechanisms that make simple designs possible. The first is the ability to represent interrelated data in a meaningful hierarchy. For example, devices have interfaces and interfaces have IP addresses. Conceptually this seems like a very simple structure. However, if we were to manually use the REST API or ORM to handle creating objects like this, we would first have to create a device object and keep its ID in memory. We would then have to create interfaces with their device foreign-key set to the device ID we just created. Finally, we’d have to save all of the interface IDs and do the same with IP addresses. Design Builder provides a means to represent objects in YAML and produce their representation within the Nautobot database. A typical design workflow follows the following diagram:

Design Workflow

Following this process, we can produce YAML files that intuitively represent the structure of the data we want to create. An example of a Design Builder YAML design can be seen in the following YAML document:

devices:
  - name: "Router 1"
    status__name: "Active"
    interfaces:
      - name: "GigabitEthernet0"
        type: "1000base-t"
        status__name: "Active"
        ip_addresses:
          - address: "192.168.0.1/24"
            status__name: "Active"

This YAML document would produce a single device, with a single Gigabit Ethernet interface. The interface itself has a single IP address. As demonstrated in the example, Design Builder automatically associates the parent/child relationships correctly, and there is no need to keep copies of primary and foreign keys. We can visually represent this YAML design with the following diagram:

example object

Design Builder also provides a system to query for existing related objects using some attribute of the associated object. In the above example, the status field is actually a related object. Statuses are not just simple strings, they are first-class objects within the Nautobot database. In this case, the Status object with the name Active is predefined in Nautobot and does not need to be created. It does, however, need to be associated with the Device, the Interface, and the IPAddress objects.

This object relationship is actually a foreign-key relationship in the database and ORM. If we were using the Django ORM to associate objects, we would first need to look up the status before creating the associated objects. Design Builder provides a way to perform that lookup as part of the model hierarchy. Note that we’re looking up the status by its name: status__name. Design Builder has adopted similar syntax to Django’s field lookup. The field name and related field are separated by double underscores.

Use Cases

There are many use cases that are covered by the Design Builder, but we will highlight a very simple one in this post. Our example use case handles the creation of edge site designs within Nautobot. This use case is often seen when doing tech refreshes or new site build-outs.

Engineers commonly need to add a completely new set of data for a site. This could be the result of a project to refresh a site’s network infrastructure or it could be part of deploying a new site entirely. Even with small sites, the number of objects needing to be created or updated in Nautobot could be dozens or even hundreds. However, if a standardized design is developed then Design Builder can be used to auto-populate all of the data for new or refreshed sites.

Consider the following design, which will create a new site with edge routers, a single /24 prefix and two circuits for the site:

---
sites:
  - name: "LWM1"
    status__name: "Staging"
    prefixes:
      - prefix: "10.37.27.0/24"
        status__name: "Reserved"
    devices:
      - name: "LWM1-LR1"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
      - name: "LWM1-LR2"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"      
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"

circuits:
  - cid: "LWM1-CKT-1"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"

  - cid: "LWM1-CKT-2"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"

This is still quite a bit of information to write. Luckily, the Design Builder application can consume Jinja templates to produce the design files. Using some Jinja templating, we can reduce the above design a bit:


---
sites:
  - name: "LWM1"
    status__name: "Staging"
    prefixes:
      - prefix: "10.37.27.0/24"
        status__name: "Reserved"
    devices:
    {% for i in range(2) %}
      - name: "LWM1-LR{{ i }}"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
    {% endfor %}
circuits:
  {% for i in range(2) %}
  - cid: "LWM1-CKT-{{ i }}"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"
  {% endfor %}

The above design file gets closer to a re-usable design. It has reduced the amount of information we have to represent by leveraging Jinja2 control structures, but there is still statically defined information. At the moment, the design includes hard coded site information (for the site name, device names and circuit IDs) as well as a hard coded IP prefix. Design Builder also provides a way for this information to be gathered dynamically. Fundamentally, all designs are just Nautobot Jobs. Therefore, a design Job can include user-supplied vars that are then copied into the Jinja2 render context. Consider the design job for our edge site design:

class EdgeDesign(DesignJob):
    """A basic design for design builder."""
    site_name = StringVar(label="Site Name", regex=r"\w{3}\d+")
    site_prefix = IPNetworkVar(label="Site Prefix")

#...

This design Job collects a site_name variable as well as a site_prefix variable from the user. Users provide values for these variables through the normal Job launch entrypoint:

Edge Site Job Input Variables

Once the job has been launched, the Design Builder will provide these input variables to the Jinja rendering context. The variable names, within the jinja2 template, will match the attribute names used in the Design Job class. With the site_name and site_prefix variables now being defined dynamically, we can produce a final design document using them:

---

sites:
  - name: "{{ site_name }}"
    status__name: "Staging"
    prefixes:
      - prefix: "{{ site_prefix }}"
        status__name: "Reserved"
    devices:
    {% for i in range(2) %}
      - name: "{{ site_name }}-LR{{ i }}"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
    {% endfor %}
circuits:
  {% for i in range(2) %}
  - cid: "{{ site_name }}-CKT-{{ i }}"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "{{ site_name }}"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"
  {% endfor %}

The design render context is actually much more flexible than simple user entry via script vars. Design Builder provides a complete system for managing the render context, including loading variables from YAML files and providing dynamic content via Python code. The official documentation covers all of the capabilities of the design context.

In addition to the YAML rendering capabilities, Design Builder includes a way to perform just-in-time operations while creating and updating Nautobot objects. For instance, in the above example, the site prefix is specified by the user that launches the job. It may be desirable for this prefix to be auto-assigned and provisioned out of a larger parent prefix. Design Builder provides a means to perform these just-in-time lookups and calculations in the form of something called an “action tag”. Action tags are evaluated during the object creation phase of a design’s implementation. That means that database lookups can occur and computations can take place as the design is being implemented. One of the provided action tags is the next_prefix action tag. This tag accepts query parameters to find a parent prefix, and also a parameter that specifies the length of the required new prefix. For example, if we want to provision a /24 prefix from the 10.0.0.0/16 parent, we could use the following:

prefixes:
  - "!next_prefix":
      prefix: "10.0.0.0/16"
      length: 24
    status__name: "Active"

The next_prefix action tag will find the parent prefix 10.0.00/16 and look for the first available /24 in that parent. Once found, Design Builder will create that child prefix with the status Active.

Several action tags are provided out of the box, but one of the most powerful features of Design Builder is the ability to include custom action tags in a design. Action tags are implemented in Python as specialized classes, and can perform any operation necessary to produce a just-in-time result.

There is quite a lot to understand with Design Builder, and we have only touched on a few of its capabilities. While there are several moving parts, the following diagram illustrates the high-level process that the Design Builder application uses to go from design files and templates to an implemented design.

Design Builder Process

Design Builder starts with some optional input variables from the Nautobot job and combines them with optional context variables written either in YAML or Python or both. This render context is used by the Jinja2 renderer to resolve variable names in Jinja2 templates. The Jinja2 templates are rendered into YAML documents that are unmarshaled as Python dictionaries and provided to the Builder. The Builder iterates all of the objects in this dictionary and performs necessary database creations and updates. In the process of creating and updating objects, any action tags that are present are evaluated. The final result is a set of objects in Nautobot that have been created or updated by Design Builder.

Roadmap

Our plans for Design Builder are far from over. There are many more features we’re currently working on, as well as some that are still in the planning stages. Some of the near-term features include design lifecycle and object protection.

The design lifecycle feature allows the implementations of a design to be tracked. Design instances can be created (such as an instance of the edge site design above) and can be subsequently decommissioned. Objects that belong to a design instance will be reverted to their state prior to the design implementation, or they may be removed entirely (if created specifically for a design). Designs can also track inter-design dependencies so that a design cannot be decommissioned if other design instances depend on it. The design lifecycle feature will also allow designs to be versioned so that an implementation can be updated over time.

The ability to protect objects that belong to a design is also planned. The idea is that if an object is created as part of a design implementation, any attributes that were initially set in this design cannot be updated outside of that design’s lifecycle. This object protection assures that our source of truth has data that complies with a design and prevents manually introduced errors.

Conclusion

Design Builder is a great tool that ensures your network designs are used for every deployment, and simplifies populating data in Nautobot along the way. It provides a streamlined way to represent hierarchical relationships with a clear syntax and concepts that should be familiar to those that have started to embark on their NetDevOps journey. I encourage you to try it out.

-Andrew, Christian and Paddy

]]>
Andrew Bates, Christian Adell, Paddy Kelly
Nautobot and Django QuerySet Annotations - Part 32024-01-18T00:00:00+00:002024-01-18T00:00:00+00:00https://blog.networktocode.com/post/nautobot-and-django-query-annotations-part-3This is Part 3 of the series on Django QuerySet annotations. In Part 1 we talked about what the annotations are and why you would want to use them. In Part 2 we looked at subqueries and more annotation examples.

In this post, we’ll learn how to use annotations spanning multiple tables, and we will go through more advanced use cases.

In the first part of this series we looked at examples of using Count aggregator with annotations to count the number of related objects. We took advantage of reverse relationships1 created by Django and used value of the related_name argument for accessing related objects.

However, not all related objects have a reverse relationship. When a model is defined, it’s possible to explicitly disable the reverse relationship by using the value + (plus sign) for the related_name. Or perhaps our models are indirectly related, i.e., there is no ForeignKey pointing directly from one model to another.

In these cases we have to use subqueries2 in our annotations to process related objects. The resulting queries tend to be a bit more complex; but once you get the hang of it, it all starts to make sense.

Let’s go back to the example where we counted number of circuits for each provider. We’ll try to build an equivalent query that doesn’t use the explicit relationship but uses a subquery instead. This subquery will leverage the fact that the Circuit model has ForeignKey pointing to the Provider model.

This is the query we used previously:

from django.db.models import Count

providers = Provider.objects.annotate(circuit_count=Count("circuits"))

Let’s modify it. First, we build the skeleton of the outer query:

from django.db.models import Subquery, OuterRef

Provider.objects.annotate(circuit_count=Subquery(...))

We keep the same annotation field name but instead of using the Count aggregator we insert a result returned by the subquery.

We now need to focus on building that subquery:

circuits = Circuit.objects \
                  .filter(provider=OuterRef("pk")) \
                  .values("provider") \
                  .order_by() \
                  .annotate(c=Count("provider")) \
                  .values("c")

That’s a bit more complex than using Count aggregator, but sometimes this is the only available option.

Before we combine the subquery and the outer query to get our final result, let’s have a closer look at all of the components of the subquerry to make sure we understand what they do.

  • filter(provider=OuterRef("pk")) — We want only Circuit objects with provider equal to the provider in the outer query. Field pk in OuterRef3 refers to the primary key of the Provider model.
  • values("provider") — Using values clause before annotation will trigger grouping behavior of the annotation as discussed in Part 2 of this series. We will count number of circuits for given provider, so we specify provider field here.
  • order_by() — We clear the default ordering to ensure grouping operation returns correct results.
  • annotate(c=Count("provider")) — Now we apply our annotation with the Count aggregator. New field, named c, will hold number of circuits for each provider group. We applied filtering at the beginning, so there will be only one such group.
  • values("c") — Finally, we specify that only one column is returned, which is the column added by the annotation.

Our subquery must return only one row and one column so that it can be used in the subsequent annotation. This is why we had to use values("c"); otherwise two columns would be returned, and we would get an error.

In this particular subquery only one row will be returned because of the filtering we applied. That won’t always be the case, so we will talk later about how to explicitly force subquery to return one row only.

We built the subquery and talked through how it works. We now need to use it in the main query to get circuit count for each of the providers:

providers = Provider.objects \
                    .annotate(circuit_count=Subquery(circuits)) \
                    .values("name", "circuit_count")

That’s it. This should have done the job. Let’s investigate the resulting values:

>>> pprint(list(providers))
[{'circuit_count': None, 'name': 'AT&T'},
 {'circuit_count': None, 'name': 'Cogent'},
 {'circuit_count': None, 'name': 'Deutsche Telekom'},
 {'circuit_count': None, 'name': 'GTT'},
 {'circuit_count': None, 'name': 'Liberty Global'},
 {'circuit_count': None, 'name': 'Lumen Technologies'},
 {'circuit_count': 40, 'name': 'NTT'},
 {'circuit_count': None, 'name': 'Orange'},
 {'circuit_count': None, 'name': 'PCCW Global'},
 {'circuit_count': None, 'name': 'Sprint'},
 {'circuit_count': None, 'name': 'Tata Communications'},
 {'circuit_count': None, 'name': 'Telecom Italia'},
 {'circuit_count': 40, 'name': 'Telia Carrier'},
 {'circuit_count': None, 'name': 'Telxius'},
 {'circuit_count': None, 'name': 'Verizon'},
 {'circuit_count': None, 'name': 'Zayo'}]

Something is not quite right here. Why is None showing up for some of the providers? When we used Count with reverse relationships, the value 0 was shown for the providers that don’t have circuits.

We get None here because when the subquery doesn’t return any results for given provider, a value of None will be used when annotating the Provider objects.

Fortunately, Django provides a function that can help us fix this. We’ll look at it now.

Coalesce Function

We saw that when subquery does not return anything, the value assigned to the annotation field is None. This is generally not what we want. If subquery doesn’t return anything, we would like to define a default value that is assigned to a field instead.

It so happens that Django provides a function that does exactly that. It’s a function named Coalesce4. Imagine it working like a dictionary get() method in Python, with default return value provided.

Coalesce accepts a list of at least two field names or expressions and returns the first non-null value. This means we could use multiple subqueries and wrap them in the Coalesce function. The first non-empty result from the ones returned by subqueries will be used. The caveat here is that the arguments must be of similar type. You can’t mix numbers with text, for example.

So now we know we have to use Coalesce function to fix our query. Let’s go ahead and feed it the subquery we created and an integer 0, which will be our default value.

from django.db.models.functions import Coalesce

providers = Provider.objects \
                    .annotate(
                        circuit_count=Coalesce(Subquery(circuits), 0)
                    ) \
                    .values("name", "circuit_count")

Let’s investigate the resulting values:

>>> pprint(list(providers))
[{'circuit_count': 0, 'name': 'AT&T'},
 {'circuit_count': 0, 'name': 'Cogent'},
 {'circuit_count': 0, 'name': 'Deutsche Telekom'},
 {'circuit_count': 0, 'name': 'GTT'},
 {'circuit_count': 0, 'name': 'Liberty Global'},
 {'circuit_count': 0, 'name': 'Lumen Technologies'},
 {'circuit_count': 40, 'name': 'NTT'},
 {'circuit_count': 0, 'name': 'Orange'},
 {'circuit_count': 0, 'name': 'PCCW Global'},
 {'circuit_count': 0, 'name': 'Sprint'},
 {'circuit_count': 0, 'name': 'Tata Communications'},
 {'circuit_count': 0, 'name': 'Telecom Italia'},
 {'circuit_count': 40, 'name': 'Telia Carrier'},
 {'circuit_count': 0, 'name': 'Telxius'},
 {'circuit_count': 0, 'name': 'Verizon'},
 {'circuit_count': 0, 'name': 'Zayo'}]

Great! We fixed our problem, and now providers that don’t have any circuits have the value of the field circuit_count set to 0.

Limiting Subquery Results to One Column and One Row

As I mentioned in the previous example, a subquery will often return more than one row and column, which cannot be used in some annotations.

Consider the below example of annotating Location objects with the name of the most recently added device within each location:

recently_added = Device.objects \
                       .filter(location=OuterRef("pk")) \
                       .order_by("-last_updated") \
                       .values("name")

locations = Location.objects \
                    .annotate(newest_device=Subquery(recently_added))
>>> print(locations)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  ... (cut for brevity)
django.db.utils.ProgrammingError: more than one row returned by a subquery used as an expression

We defined subquery recently_added, which returns devices located in a given location, ordered by the date they were last updated.

When we try to use this subquery in an annotation, we get an error. Django helpfully tells us that subquery returned too many rows. This happens because there could be multiple devices located in any given location, and each row contains device record.

To fix this, we use one element slice [:1] on the QuerySet returned by the subquery. Subquery expects to receive a QuerySet that produces one row, which is why we are using a slice. Note that we cannot use [0] index or first() function here, as these return a single instance of an object, or a dictionary when used with values.

recently_added = Device.objects \
                       .filter(location=OuterRef("pk")) \
                       .order_by("-last_updated") \
                       .values("name")[:1]

locations = Location.objects \
                    .annotate(newest_device=Subquery(recently_added)) \
                    .values("name", "newest_device")
>>> pprint(list(locations))
[{'name': 'Jersey City', 'newest_device': 'jcy-spine-02.infra.ntc.com'},
 {'name': 'New York City', 'newest_device': 'nyc-spine-02.infra.ntc.com'},
 {'name': 'ams', 'newest_device': 'ams-leaf-08'},
 {'name': 'atl', 'newest_device': 'atl-leaf-08'},
 {'name': 'bkk', 'newest_device': 'bkk-leaf-08'},
 {'name': 'can', 'newest_device': 'can-leaf-08'},
 {'name': 'cdg', 'newest_device': 'cdg-leaf-08'},
 {'name': 'del', 'newest_device': 'del-leaf-10'},
 {'name': 'den', 'newest_device': 'den-leaf-08'},
 {'name': 'dfw', 'newest_device': 'dfw-leaf-08'},
 {'name': 'dxb', 'newest_device': 'dxb-leaf-08'},
 {'name': 'fra', 'newest_device': 'fra-leaf-08'},
 {'name': 'hkg', 'newest_device': 'hkg-leaf-08'},
 {'name': 'hnd', 'newest_device': 'hnd-leaf-08'},
 {'name': 'icn', 'newest_device': 'icn-leaf-04'},
 {'name': 'jfk', 'newest_device': 'jfk-leaf-08'},
 {'name': 'lax', 'newest_device': 'lax-leaf-10'},
 {'name': 'lhr', 'newest_device': 'lhr-leaf-08'},
 {'name': 'ord', 'newest_device': 'ord-leaf-08'},
 {'name': 'pek', 'newest_device': 'pek-leaf-08'},
 {'name': 'pvg', 'newest_device': 'pvg-leaf-04'},
 {'name': 'sin', 'newest_device': 'sin-leaf-08'}]

And there you have it — our annotation is now working as expected.

Annotations Across Many Tables

A word of warning when using annotations across more than two tables: Due to the way Django translates queries with annotations to underlying SQL, using aggregator functions like Count or Avg across multiple tables will give incorrect results.

For instance, if we try annotating Location records with the count of related prefixes and the count of related devices, we get incorrect numbers:

>>> locations = Location.objects.annotate(pfx_count=Count("prefixes"))
>>> locations.get(name="lax").pfx_count
5
>>> locations = Location.objects.annotate(device_count=Count("devices"))
>>> locations.get(name="lax").device_count
12
>>> locations = Location.objects.annotate(pfx_count=Count("prefixes"), device_count=Count("devices"))
>>> locations.get(name="lax").device_count
60
>>> locations.get(name="lax").pfx_count
60

As you can see above, when using a single annotation with Count, we get correct results. But when we try to annotate Location objects with related Prefix and Device objects in a single query, we get back an incorrect value for both fields5.

Note that the in the case of Count, we can address this problem by providing argument distinct=True when invoking Count. However, this will not work for other aggregation functions.

>>> locations = Location.objects \
                        .annotate(
                            pfx_count=Count("prefixes", distinct=True),
                            device_count=Count("devices", distinct=True)
                        )
>>> locations.get(name="lax").device_count
12
>>> locations.get(name="lax").pfx_count
5

We will see how we can solve this problem for the generic case by using our trusty subqueries. In the example below, each of the subqueries will be run independently behind the scenes, and the results of each will be assigned to the given field.

from django.db.models import Subquery, OuterRef

pfx_count_sq = Location.objects \
                       .filter(pk=OuterRef("pk")) \
                       .annotate(c=Count("prefixes")) \
                       .values("c")

device_count_sq = Location.objects \
                          .filter(pk=OuterRef("pk")) \
                          .annotate(c=Count("devices")) \
                          .values("c")

locations = Location.objects \
                    .annotate(
                        pfx_count=Subquery(pfx_count_sq), \
                        device_count=Subquery(device_count_sq) \
                    )
>>> locations.get(name="lax").pfx_count
5
>>> locations.get(name="lax").device_count
12

First, we defined two subqueries, pfx_count_sq and device_count_sq. These could be included directly inside of the annotate() method but defining them separately makes the code easier to read.

In each of the subqueries, we used OuterRef class, which allows us to use the pk field from the outer Location object when Subquery is run.

Each subquery uses annotation with a Count aggregator. We use values("c") here to return just one column c, otherwise we could not use the result of the subquery in the outer query annotation.

You also need to ensure only one row is returned. In our case, we are guaranteed to have only one match. If your subquery returns more than one row, you’d have to use slice [:1] to get back only one item.

Lastly, we wrap subqueries in SubQuery class and plug them into the annotate() method in the main query to get the final result.

As you can see, the resulting values now match the numbers we got when using two separate Count annotations.

Note that using Count and other Django aggregation functions is usually more performant than using subqueries because of the underlying database optimizations. However, subqueries are generally preferable to expressing the equivalent logic in Python code. In case of complex subqueries, you should profile underlying database queries to ensure their performance is acceptable.

More Examples

We’ll now have a look at some more examples of annotations, using different Django functions.

Using Length Function

Let’s compute the length of each of the device names and sort the results in descending order.

from django.db.models import F
from django.db.models.functions import Length

devices = Device.objects \
                .annotate(name_len=Length(F("name"))) \
                .order_by("-name_len")
>>> pprint(list(devices)[:20])
[<Device: jcy-spine-02.infra.ntc.com>,
 <Device: nyc-spine-01.infra.ntc.com>,
 <Device: jcy-spine-01.infra.ntc.com>,
 <Device: nyc-spine-02.infra.ntc.com>,
 <Device: nyc-leaf-02.infra.ntc.com>,
 <Device: nyc-leaf-01.infra.ntc.com>,
 <Device: jcy-rtr-01.infra.ntc.com>,
 <Device: nyc-rtr-01.infra.ntc.com>,
 <Device: jcy-rtr-02.infra.ntc.com>,
 <Device: nyc-rtr-02.infra.ntc.com>,
 <Device: jcy-bb-01.infra.ntc.com>,
 <Device: nyc-bb-01.infra.ntc.com>,
 <Device: den-leaf-02>,
 <Device: dfw-leaf-05>,
 <Device: dxb-leaf-02>,
 <Device: dxb-leaf-05>,
 <Device: fra-leaf-04>,
 <Device: fra-leaf-08>,
 <Device: hkg-edge-01>,
 <Device: hkg-leaf-06>]

We made use of Length6 function and F7 object. F object is used to pass value of the device’s name field to the Length function. Length function will return number of characters in the passed argument.

Filtering Values of the Annotation Field

In the below example, we use Count in the annotation. Then, we filter the values in the annotation field vlans_count to get back only locations that have at least one VLAN assigned to them.

locations = Location.objects \
                    .annotate(vlans_count=Count("vlans")) \
                    .filter(vlans_count__gt=1) \
                    .values("name", "vlans_count")
>>> pprint(list(locations))
[{'name': 'ams', 'vlans_count': 16},
 {'name': 'atl', 'vlans_count': 16},
 {'name': 'bkk', 'vlans_count': 16},
 {'name': 'can', 'vlans_count': 16},
 {'name': 'cdg', 'vlans_count': 16},
 {'name': 'del', 'vlans_count': 20},
 {'name': 'den', 'vlans_count': 16},
 {'name': 'dfw', 'vlans_count': 16},
 {'name': 'dxb', 'vlans_count': 16},
 {'name': 'fra', 'vlans_count': 16},
 {'name': 'hkg', 'vlans_count': 16},
 {'name': 'hnd', 'vlans_count': 16},
 {'name': 'icn', 'vlans_count': 8},
 {'name': 'jfk', 'vlans_count': 16},
 {'name': 'lax', 'vlans_count': 20},
 {'name': 'lhr', 'vlans_count': 16},
 {'name': 'ord', 'vlans_count': 16},
 {'name': 'pek', 'vlans_count': 16},
 {'name': 'pvg', 'vlans_count': 8},
 {'name': 'sin', 'vlans_count': 16}]

Using Exists Subquery

We can use Exists8 subquery subclass to get back a boolean True if subquery returns results or False if it doesn’t. For instance, we might want to check whether a given location has devices that are assigned edge role.

from django.db.models import Exists, OuterRef

edge_dev_sq = Subquery(
                Device.objects \
                      .filter(
                          location=OuterRef("pk"),
                          device_role__slug="edge"
                      )
              )

locs_w_edge_dev = Location.objects \
                          .annotate(has_edge_devices=Exists(edge_dev_sq)) \
                          .values("name", "has_edge_devices")
>>> pprint(list(locs_w_edge_dev))
[{'has_edge_devices': False, 'name': 'Jersey City'},
 {'has_edge_devices': False, 'name': 'New York City'},
 {'has_edge_devices': True, 'name': 'ams'},
 {'has_edge_devices': True, 'name': 'atl'},
 {'has_edge_devices': True, 'name': 'bkk'},
 {'has_edge_devices': True, 'name': 'can'},
 {'has_edge_devices': True, 'name': 'cdg'},
 {'has_edge_devices': True, 'name': 'del'},
 {'has_edge_devices': True, 'name': 'den'},
 {'has_edge_devices': True, 'name': 'dfw'},
 {'has_edge_devices': True, 'name': 'dxb'},
 {'has_edge_devices': True, 'name': 'fra'},
 {'has_edge_devices': True, 'name': 'hkg'},
 {'has_edge_devices': True, 'name': 'hnd'},
 {'has_edge_devices': True, 'name': 'icn'},
 {'has_edge_devices': True, 'name': 'jfk'},
 {'has_edge_devices': True, 'name': 'lax'},
 {'has_edge_devices': True, 'name': 'lhr'},
 {'has_edge_devices': True, 'name': 'ord'},
 {'has_edge_devices': True, 'name': 'pek'},
 {'has_edge_devices': True, 'name': 'pvg'},
 {'has_edge_devices': True, 'name': 'sin'}]

Using Python Expressions in Annotations

You can also use Python expressions in your annotations. We’ve seen an example of that in Part 2, where we computed percentage of free interfaces on devices. Here is an example that shows how to annotate Device objects with the time elapsed since the object was created.

import datetime

today = datetime.date.today()

devices = Device.objects \
                .annotate(
                    created_delta=today - F("created")
                )
>>> devices[0].created_delta
datetime.timedelta(days=47)
>>> devices[0].created_delta.days
47

We use datetime.datetime.date.today() to get today’s date. Then, we use this value in the annotation to subtract the date the object was created from today’s date. The result is a datetime.timedelta object, which we can query for the number of days since the device was created.

Count distinct Argument

In our final example we will use Count’s distinct argument to compute number of distinct device roles used by devices in each location.

When we use distinct=True, we’re telling Count to compute number of unique values for the given argument. Here, we’re giving it devices__device_role, which will access device role for the devices in each of the locations.

locations = Location.objects \
                    .annotate(
                         devrole_count=Count(
                            "devices__device_role", distinct=True
                         )
                    ) \
                    .values("name", "devrole_count")
>>> pprint(list(locations))
[{'devrole_count': 3, 'name': 'Jersey City'},
 {'devrole_count': 4, 'name': 'New York City'},
 {'devrole_count': 2, 'name': 'ams'},
 {'devrole_count': 2, 'name': 'atl'},
 {'devrole_count': 2, 'name': 'bkk'},
 {'devrole_count': 2, 'name': 'can'},
 {'devrole_count': 2, 'name': 'cdg'},
 {'devrole_count': 2, 'name': 'del'},
 {'devrole_count': 2, 'name': 'den'},
 {'devrole_count': 2, 'name': 'dfw'},
 {'devrole_count': 2, 'name': 'dxb'},
 {'devrole_count': 2, 'name': 'fra'},
 {'devrole_count': 2, 'name': 'hkg'},
 {'devrole_count': 2, 'name': 'hnd'},
 {'devrole_count': 2, 'name': 'icn'},
 {'devrole_count': 2, 'name': 'jfk'},
 {'devrole_count': 2, 'name': 'lax'},
 {'devrole_count': 2, 'name': 'lhr'},
 {'devrole_count': 2, 'name': 'ord'},
 {'devrole_count': 2, 'name': 'pek'},
 {'devrole_count': 2, 'name': 'pvg'},
 {'devrole_count': 2, 'name': 'sin'}]

Conclusion

This post concludes the series on Nautobot and Django QuerySet annotations. We learned about annotations and subqueries and their use cases. We explored different functions that can be used with annotations and worked through a number of practical examples which showcase their power.

With this acquired knowledge, you should be able to write more powerful and efficient queries.

I hope you enjoyed this series and are looking forward to writing some awesome Nautobot code!

-Przemek Rogala

References

]]>
Przemek Rogala
Introducing Cookiecutter Project Templates to Support Nautobot App Development for Network Automation2024-01-10T00:00:00+00:002024-01-10T00:00:00+00:00https://blog.networktocode.com/post/introducing-cookiecutter-nautobot-appIn June of 2022 Network to Code announced the opensourcing of a Cookiecutter project that contains a project template ChatOps integrations to help in the initial bootstrapping of a Nautobot ChatOps App. The Nautobot ChatOps Cookiecutter announcement post can be found here. This first cookie helped to lower the barrier for entry when it came to developing new ChatOps integrations AND made it possible to have baseline standards on things like local development environments, docs structure (with docs already built for installation), administration, and contributing.

Two New Cookies and a New Home for Another

Today we are announcing that we are doubling down on the benefits of Cookiecutter and are opensourcing a new Cookiecutter repository that contains three separate cookies. Two of the cookies are new to open sourcing and the original ChatOps cookie is getting a new home within this same repository!

Nautobot App

Nautobot App cookie is broadly applicable to most Nautobot Apps you may develop and provides the initial scaffolding for building additional models for your Nautobot App. This cookie provides a question when baking the cookie for a Model Class Name. If provided, a simple model class will be created with name and description fields on the model along with all standard components, to have a fully functioning set of UI and API views.

Nautobot App SSoT

Nautobot App SSoT is a superset of the Nautobot App cookie, as it provides the same capability to automatically generate a model with all required components in order to have a fully functioning set of UI and API views to support a network source of truth. In addition to features provided by Nautobot Apps cookie, this cookie will also build out the Network to Code recommended folder and file structure for developing an SSoT App as well as the required Nautobot SSoT Jobs (creates both Data Source and Data Target Jobs). This includes the initial creation of DiffSync adapters, models, and utils along with their use in the Nautobot SSoT Jobs.

Nautobot App ChatOps

Nautobot App ChatOps cookie is also a superset of the Nautobot App cookie but instead provides a base command and a hello_world subcommand along with the required settings in the pyproject.toml that are used to inform the Nautobot ChatOps App that an additional base command is registered as part of this app. This cookie was previously open sourced as its own Cookiecutter repository but has now been migrated to the new repository, and the old repository has been archived.

Why the New Repository?

As the amount of open source projects Network to Code maintains continues to expand, we are evaluating how to be as effective as possible in the care and feeding that is required for maintaining any open source project. This also is factoring in things like continual maintenance of standards when it comes to docs structure, how to interface with development environments, and CI workflow standards. With the new repository, we are able to provide the same standard for three separate cookies and the use of symbolic links. This allows a change to a single file to immediately be applicable to all cookies in the repository. By providing this functionality we are able to avoid drift between all cookies, as the symbolic links all point back to one overarching standard!

Codified Standards

With the open sourcing of more cookies for Nautobot Apps, Network to Code has also built an internal Drift Management process that is already helping to keep Nautobot Apps maintained by Network to Code all at the same standard, no matter when the cookie was baked. This helps to improve developer experience across all of our Official Nautobot Apps and ensure consistent standards for testing, basic docs, and CI!

-Jeremy White

]]>
Jeremy White
Last Month in Nautobot - December 20232024-01-08T00:00:00+00:002024-01-08T00:00:00+00:00https://blog.networktocode.com/post/last-month-in-nautobot-2312Welcome to our monthly Nautobot community update! We’ll dive into the latest updates across the Nautobot community each month, highlighting the key milestones, releases, and noteworthy contributions. From new features and enhancements to bug fixes and events, there’s always something happening in our dynamic ecosystem. Open source is at the core of our values, empowering individuals and organizations to collaborate, innovate, and make a positive impact together. This monthly blog post is our way of celebrating the accomplishments and contributions of our Nautobot community members.

Nautobot Core

Release v2.1.0

v2.1.0 - 2023-12-22

Django Admin Log Entries (#4646)

Django Admin Log Entries record administrative changes made under the “Admin” section of the user interface. Changes (add/delete/update) to Objects like Users, Group, Object Permissions, etc. in the “Admin” user interface are now displayed as “Log Entries” under the “Administration” section of the Admin UI.

External Integration Model (#4694)

A new ExternalIntegration model has been added which provides a centralized store for data such as URLs and credentials that are used to access systems external to Nautobot. This information can then be used by jobs or apps to perform actions such as creating DNS records or updating configuration management tickets.

Home Page Panels Can Be Customized (#2149)

The panels displayed on the Nautobot home page have been modified to enable a more personalized user experience. Individual panels can now be collapsed, hiding the contents from view. Additionally, panels can be reordered by dragging and dropping the panels to the desired order.

Job File Outputs (#3352, #4820)

The Job base class now includes a create_file(filename, content) method which can be called by a Job to create a persistent file with the provided content when run. This file will be linked from the Job Result detail view for subsequent downloading by users and can also be downloaded via the REST API (/api/extras/file-proxies/<id>/download/) as desired.

The size of files Jobs can create via this method is constrained by the JOB_CREATE_FILE_MAX_SIZE settings variable.

Job JSONVar inputs (#4926)

Provides the ability to have native JSON data inputs for Jobs; this is provided by a multi-line text input on the Job form. And the provided JSON data is serialized prior to passing to the run() method of the Job.

UI/API isnull Filter on Nullable Fields (#1905)

Models with nullable fields (i.e., model fields with null=True) can now be filtered in the UI and the API with <field>__isnull=true/false filters. These filters are automatically added to all appropriate fields.

Data Exports as a System Job (#4745)

The data export functionality in all object list views (allowing export of all objects or a filtered subset of objects to CSV, YAML, and/or as defined by an ExportTemplate) has been changed from a synchronous operation to an asynchronous background task, leveraging the new ExportObjectList system Job. As a result, exports of thousands of objects in a single operation will no longer fail due to browser timeout.

Nautobot UI Reskin (#4677, #4765)

The Nautobot UI has been updated with a customized theme, giving it a brand-new look. In addition, Nautobot’s navigation bar has been moved from the top to the left.

Drop Support for Legacy PostgreSQL Versions (#4757)

Support for versions of PostgreSQL prior to 12.0 has been removed, as these versions are no longer maintained and contain bugs that prevent migrations from running in certain scenarios. The nautobot-server migrate or nautobot-server post_upgrade commands will now abort when detecting an unsupported PostgreSQL version.

Remove HIDE_RESTRICTED_UI Toggle (#4787)

Support for HIDE_RESTRICTED_UI has been removed. UI elements requiring specific permissions will now always be hidden from users lacking those permissions. Additionally, users not logged in will now be automatically redirected to the login page.

Releases - LTM 1.6

  • v1.6.8 - 2023-12-22
    • #4876 - Updated cryptography to 41.0.7 due to CVE-2023-49083. As this is not a direct dependency of Nautobot, it will not auto-update when upgrading. Please be sure to upgrade your local environment.
    • #4988 - Fixed missing object-level permissions enforcement when running a JobButton (GHSA-vf5m-xrhm-v999)
    • #4988 - Removed the requirement for users to have both extras.run_job and extras.run_jobbutton permissions to run a Job via a Job Button. Only extras.run_job permission is now required.
    • #5002 - Updated paramiko to 3.4.0 due to CVE-2023-48795. As this is not a direct dependency of Nautobot, it will not auto-update when upgrading. Please be sure to upgrade your local environment.
    • Bug fixes
  • v1.6.7 - 2023-12-12

Apps Ecosystem

Community

Get in Touch!

Do you have any cool Nautobot-related project we should write about? Swing by the Network to Code Slack -> channel #nautobot and write us a quick line! Sign up here if you don’t have an account.

-Cristian and Gary

]]>
Cristian Sirbu and Gary Snider
Introduction to Event-Driven Ansible and Nautobot2024-01-02T00:00:00+00:002024-01-02T00:00:00+00:00https://blog.networktocode.com/post/event-driven-ansibleAt Network to Code, we are continually working on new solutions to extend automation capabilities for our customers. One project that I recently worked on used Event-Driven Ansible, or EDA, to simplify the process of automating other systems based on changes in Nautobot. This blog post will cover the basics of EDA, and how we used it to update ServiceNow CMDB records based on changes in Nautobot.

What Was the Problem We Were Trying to Solve?

The customer is using ServiceNow as their CMDB and Nautobot as their source of truth for network infrastructure. They wanted to be able to update ServiceNow records when changes were made in Nautobot. For example, when a device is added to Nautobot, they wanted to create a corresponding record in ServiceNow. There are other systems that we are integrating with Nautobot using EDA, but for this blog post we will focus on ServiceNow. Any system with an API or Ansible Galaxy role/collection can be integrated with Nautobot using EDA.

What Is Event-Driven Ansible?

Event-Driven Ansible was developed by Red Hat to allow listening to events from various sources and then taking action on those events using Rulebooks to define three components — sources, rules, and actions.

  • Sources — where the events are coming from. This can be a webhook, Kafka, Azure Service Bus, or other sources.
  • Rules — define the conditions that must be met for an action to be taken.
  • Actions — an action is commonly running a local playbook, but could also be generating an event, running a job template in AAP, or other actions.

How Did We Use EDA to Update ServiceNow Based on an Event from Nautobot?

We developed a small custom plugin for Nautobot that utilizes Nautobot Job Hooks to publish events to an Azure Service Bus queue. An added benefit to using ASB as our event bus was that Event-Driven Ansible already had a source listener plugin built for ASB, so no additional work was needed! See event source plugins. This allows us to initiate the connection to Azure Service Bus from Nautobot and then send events to Azure Service Bus when changes are made in Nautobot.

The flow of events is as follows:

  1. Nautobot device create (or update, delete) triggers a Job Hook.
  2. A Nautobot App publishes the event to Azure Service Bus queue. This App receives the Job Hook event from Nautobot and publishes the payload to the defined Azure Service Bus queue.
  3. Ansible EDA source plugin connects and subscribes to the Azure Service Bus queue and listens for events.
  4. EDA runs Ansible playbook to update ServiceNow.

What Do the Rulebooks and Playbooks Look Like?

Below is an example of a basic rulebook we are using. This rulebook will run the playbook add_device_to_servicenow.yml when a device is created in Nautobot.

Rulebook

---
- name: "LISTEN TO ASB QUEUE"
    hosts: localhost
    sources:
      - ansible.eda.azure_service_bug:
          connection_string: ""
          queue_name: ""

    rules:
      - name: "ADD DEVICE TO SERVICENOW"
        condition: "event.body.data.action =='create'"
        action:
          run_playbook:
            name: "add_device_to_servicenow.yml"
            verbosity: 1

You can add different sources, conditions, and rules as needed. Any information that you can extract from the event can be used in the condition.

Playbook

---
- name: "ADD DEVICE TO SERVICENOW"
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: "ADD DEVICE TO SERVICENOW"
      servicenow.servicenow.snow_record:
        state: present
        table: "cmdb_ci_netgear"
        instance: ""
        username: ""
        password: ""
        name: ""
        description: ""
        serial_number: ""
        model_id: ""
        manufacturer_id: ""

Playbooks are structured as normal, with the addition of the event variable. This variable contains the event data that was sent from Nautobot. In this example, we are using the event.body.data to extract the device name, description, serial number, platform, and manufacturer.

In the above example, we used the ServiceNow Ansible Collection to update ServiceNow. You can use any Ansible module, role, or collection to update the system you are integrating with Nautobot. One of the systems I was updating did not have an Ansible module, so I used the uri module to make API calls to the system.

Conclusion

Event-Driven Ansible is a powerful tool that can be used to integrate Nautobot with other systems. It can solve the very real problem of keeping multiple systems in sync and can be used to automate many different tasks. Feel free to join us at the Network to Code Slack channel to discuss this and other automation topics.

Resources

-Susan

]]>
Susan Hooks
Nautobot Plugins Are Now Called Nautobot Apps2023-12-27T00:00:00+00:002023-12-27T00:00:00+00:00https://blog.networktocode.com/post/nautobot-app-rebrandingNautobot Plugins & Nautobot Apps, both are used to describe the same extensibility feature of Nautobot. For a cohesive experience, Network to Code has decided to update all repos and docs to use Nautobot Apps.

What’s Changing

Today all Nautobot Apps that are maintained by Network to Code have had their repository renamed to replace plugin with ‘app’. Doing so will change the repository URL, and renamed repositories in GitHub will automatically be redirected to the new renamed URL. This ensures a seamless experience during the transition.

Now that all Nautobot Apps have been renamed, each Nautobot App will have its documentation updated to replace plugin with `app’. This will include the latest published docs hosted at docs.nautobot.com and docstrings within the code itself.

Do I Need to Change Anything in My Deployment?

No, Network to Code is not changing how the apps are packaged, installed, or configured.

Will I Need to Update Nautobot or Any App That’s Installed?

No, the renaming is on the repository, not on the Python package.

-Jeremy

]]>
Jeremy White
Breaking Ground - A Year of Milestones in Network Automation2023-12-21T00:00:00+00:002023-12-21T00:00:00+00:00https://blog.networktocode.com/post/breaking-ground-a-year-of-milestones-in-network-automationAs we bid farewell to another transformative year, the team at Network to Code is thrilled to share our major achievements across the three pillars of Nautobot: the core platform software, our cloud offering, and application development for the Nautobot app ecosystem. Each of these pillars promises to empower NTC clients and the community, representing another significant step toward a future in which every network benefits from network automation and NetDevOps practices. All of these strides were only possible with the unwavering support of our clients and community. As we reflect, we invite you to join us in celebrating the advancements made this year and the innovative solutions that are reshaping the landscape of network automation.

Nautobot v2

NTC’s open source platform, Nautobot, has undergone its most significant update to date and represents a substantial step forward. Nautobot v2 is the result of the collective expertise and dedication of the NTC team, with a sole focus on providing the open source community with an unrivaled network source of truth and automation platform. The latest version of Nautobot is packed with new features, including:

  • An enhanced IPAM model that introduces namespaces to allow for more nuanced use cases, such as duplicate IP addresses
  • IPAM performance improvements, especially when dealing with nested prefix hierarchies
  • Consolidated location model that enables custom hierarchies and eases management constraints
  • Data import/export improvements with standardized field names and enhanced file accessibility
  • A new generic role model replaces DeviceRole, RackRole, IPAMRole, and IPAddressRoleChoices models. New roles can be associated with one or more content types.
  • Full compatibility with our application ecosystem

Transitioning from Nautobot 1.x to 2.x is a straightforward process. The migration guide provides an easy-to-follow procedure paired with tooling to ensure that the transition is successful. Specifically, it verifies the compatibility of existing data with the data model changes and provides guidance on necessary adjustments. The result is a seamless transition with minimal administrative overhead.

Looking ahead, our future plans include a complete UI overhaul with a dedicated focus on enhancing the user experience and streamlined workflows, ensuring users can achieve their objectives efficiently and intuitively. We highly encourage users to provide feedback as we continually improve the UI.

For those interested in exploring Nautobot 2.0, there is a demo instance available at demo.nautobot.com. Feel free to experiment and make changes, as the server resets to a fresh state daily.

Nautobot Cloud

This year saw the debut of Nautobot Cloud, a revolutionary release designed to streamline the Nautobot adoption process by eliminating the administrative complexities associated with on-premise deployments. By doing so, Nautobot Cloud not only simplifies but also expedites the automation delivery process, providing every organization with the opportunity to leverage the power of NetDevOps practices. Moreover, it opens the door to maximizing value by allowing teams to concentrate on new development efforts rather than the overhead of infrastructure maintenance.

Nautobot Cloud boasts a myriad of additional benefits, leveraging its cloud-native architecture to offer:

  • Effortless creation of production-ready Nautobot instances with just a few clicks
  • The flexibility to create as many Nautobot instances as required
  • Rapid upgrades to new minor versions of Nautobot within minutes
  • A self-service marketplace facilitating the addition of applications to any Nautobot instance
  • Enterprise-grade dashboards providing comprehensive situational awareness
  • Robust APIs tailored for NetDevOps pipelines
  • Effectless capture and restoration of database snapshots
  • The creation of production-ready AWX instances
  • Generation of diagnostic reports

To discover how Nautobot Cloud can enable your team to supercharge their pursuit of Network Automation, visit https://networktocode.com/nautobot/nautobot-cloud/.

Nautobot App Ecosystem

NTC’s leading application, Golden Configuration, has been instrumental in standardizing environments since its inception. The latest update has elevated its capabilities, introducing three new features:

Configuration Remediation

  • The big news is the release of configuration remediation. This functionality generates the necessary commands to bring a device back into compliance. The application achieves this through two methods. The first leverages the open source library Heir Config to automatically generate the required commands. The second, known as custom compliance, enables users to create their Python methods for generating remediation commands. Furthermore, these methods can be applied based on individual compliance rules.

Configuration Plans

  • Configuration plans establish governance around the process of remediating configurations and generate configurations based on type. When remediation configurations are generated, an object is created containing the commands, change control IDs, URLs, and approval status. Plans offer added flexibility through scoping, allowing the generation of intended, missing, remediation, and manual (with flexibility to support Jinja2 templating) configurations.
  • Configuration plans provide the robustness to support single and bulk deployments of one or multiple features/snippets. They also provide a business the capability to schedule these configuration plan deployments.

Configuration Deployment

  • This step involves taking the configuration plans generated in the previous stage and deploying them into the environment. This allows for deployment to a single device or multiple devices. Governance features ensure that only approved plans can be deployed into the environment. Furthermore, Nornir serves as the back-end framework, offering users the flexibility to opt for sensible defaults in deployments or tailored settings to meet their specific requirements.

Interested in learning more and seeing the solution in action? We recommend watching our recent Golden Config Webinar.

Reflections on a Year of Progress

As we reflect on the milestones achieved in the past year, the Network to Code team is thrilled to highlight the remarkable progress we’ve made. The release of Nautobot v2 stands as a pivotal achievement, underscoring our dedication to providing the open source community with an unparalleled foundation for NetDevOps solutions. NTC’s venture into the cloud with Nautobot Cloud streamlines the adoption process, empowering every organization to leverage the transformative capabilities of network automation. The Golden Config update augments our most popular application in our app ecosystem with compelling new features, making it easier than ever to maintain a standardized global network.

As we celebrate these accomplishments, we express our deepest gratitude to our clients and community whose unwavering support has been instrumental in realizing these innovations. NTC warmly invites you to join us in shaping the path forward by engaging with the NTC community. Your participation is valued, no matter the form it takes.

Community Resources:

Webinars

NTC Blog

Slack

YouTube

LinkedIn

NTC GitHub

Nautobot GitHub

-Chris M

]]>
Chris Murray