The NTC Mag

Agile Planning Horizons (And how to Select Them) - An Enterprise Guide - Part 4

2021-10-18T00:00:00+00:00

Following from Parts 2 and 3 of this series which provided an overview of the Flow and Sprinting planning horizons, this article provides a closer look at the Program Increment, a Scaled Agile Planning Horizon.

Overview

Program Increment is a timebox artifact of SCALED AGILE operations. PI represents a committed Enterprise or Portfolio vision and execution plan. A PI is timebox for delivery of a Value Stream which is executed by “teams of teams” which are organized as Release Trains or Solution Trains.

Release Train - A long-lived team of Agile teams which delivers solutions in a value stream. The Release Train operates in synchronous cadence and practices regular, organized cross-communication around blocks and dependencies
Solution Train - An organized group of release trains that is focused on building large and complex solutions under a shared, over-arching business/technology mission.

Source: https://d2o2utebsixu4k.cloudfront.net/media/images/1544077632343-A-Value-Stream-can-have-many-ARTs-within-it.jpg

The Program Increment (PI) itself is a series of sprints (usually 4-6 sprints) in which multiple teams operate on a shared sprint cadence to deliver on one or more committed objectives of the PI.

The PI framework is an Enterprise-level synchronization of Agile Teams to deliver on large-scale enterprise objectives. Naturally, this requires large-scale synchronization of cadence and planning ceremonies across all teams. This includes daily or weekly syncs as needed (i.e. Scrum of Scrums and/or PO sync). This isn’t a status report. Instead, it is about communicating and coordinating about blockers/impediments, dependencies and “sequence of event” needs in order to ensure results are accomplished within the PI timebox for ALL teams in the train.

By synchronizing the PI planning and the following sprints across all Teams and Trains tasked with delivery, cross-domain planning is enabled. As a result, the Scaled Agile organization benefits from:

Structured Cadence
- Improved predictability
- Improved stakeholder confidence in wait times for committed capability
- Support for regular planning and cross-functional coordination
- Limitation of batch sizes of work to discrete intervals
- Scheduled integration points
Synchronized Cross-Domain Planning
- Improved control of injection of new work
- Improved and routine dependency management
- Supports full-system integration and assessment
- Multiple feedback perspectives
- Facilitates the ability for all stakeholders to engage with each other at the same time
- Requirements and design planning occurs in a comprehensive way
- Important stakeholder decisions are accelerated
- Trains and Teams create and take responsibility for the PI plans created
- Sequences and dependencies are understood and accounted for among Trains and Teams

The PI Planning Horizon requires a focus on dependencies and sequence of execution across teams for the purpose of achieving aligned Enterprise goals. A more detailed overview of the Program Increment can be found here: [https://www.scaledagileframework.com/program-increment/]

Ceremonies

There are a number of ceremonies which typically occur within the PI:

PI Planning - PI Planning is a Scaled Agile ceremony (usually lasting 1-3 days), in which multiple teams gather to establish, plan, and commit to the objectives to be delivered collectively by the teams. Each team will create and commit to its own objectives, but these will be in alignment with an over-arching vision and roadmap. The ceremony should have a standard agenda which starts with a presentation of the business context and vision, followed by team breakouts where each team’s plans and commitments are established and presented. A more detailed overview of this ceremony can be found here: [https://www.scaledagileframework.com/pi-planning/]]

Scrum of Scrums and PO Sync - Teams should have a built-in schedule to sync every 1 or 2 weeks (or more frequently if needed) to coordinate around various team or train dependencies and review progress and impediments to make plan adjustments as needed. This sync can occur as either a Scrum of Scrums or a PO Sync, or it can be combined into a single ceremony, as appropriate.

System Demo - The system demo is held collectively at the end of each sprint among all Teams in a Train. The objective is to demonstrate the current state of the solution overall, and gather feedback from sponsors/stakeholders/customers at the “Train Level”. This ceremony should be an integrated demonstration that provides an objective measure of functionality, progress, and ultimately value of the work delivered so far within the PI. This feedback provides an opportunity for the Train to validate its progress against PI objectives and/or to make course corrections. At the end of the last sprint in the PI, there is a “Final System Demo” which is delivered in the same way, and should be a fully integrated demonstration of the functional product achieved in the PI against the committed objectives. A more detailed review of this ceremony can be found here: [https://www.scaledagileframework.com/system-demo/]

Inspect and Adapt Workshop - The Inspect and Adapt Workshop is held at the end of the PI. It should include all of the teams and stakeholders who participated in PI Planning and/or were involved in supporting the execution of the PI. This event allows the teams and stakeholders to inspect the end result of the PI and engage in collective exercises for the purpose of continuous learning and improvement. The workshop consists of three parts:

Final System Demo
Quantitative and Qualitative Assessment
Retrospective and Problem-Solving Workshop

A more detailed review of this ceremony can be found here: [https://www.scaledagileframework.com/inspect-and-adapt/]

Artifacts

Capability (or “Solution Capability”) - A Capability is a higher-level behavior or functionality that is identified, planned, refined, and committed for delivery by the Solution Train; typically the Solution Capability is delivered by multiple Release Trains. The Capability is composed of one or more Features and is sized to fit within the PI. Capabilities are usually created by Product Managers in collaboration with Product Owners and other key stakeholders. Capabilities are structured like features, but they are at a higher level of abstraction, describing system-level behavior and function in support of the definition and development of large solutions. Solution Capabilities comprise Strategic Solutions: the products, services and systems delivered to the customer.

Master Feature - In some Scaled Agile organizations or contexts, Master Features exist to help enterprises organize and group related Features. Master Features have the same properties as Features and Capabilities (overview, statement of benefit, and acceptance criteria) but exist at a level of abstraction higher than that of a Feature but lower than that of a Capability.

Feature - A Feature is a service or functionality that fulfills a stakeholder need.

The Feature is the primary artifact, which is identified, estimated, and groomed to “ready state”, and ultimately planned and committed to by the Release Train within the sprints of a Program Increment.

The feature should include a description of the service or function as well as a statement of benefit and acceptance criteria. The Feature is composed of one or more User Stories. Features are typically created by the Product Owner, with Solution (or System) Architects commonly creating enabler features that will be required for delivery of the functional Features and Capabilities. Features are committed to by Release Trains at the PI level, and the constituent Stories are committed to by Teams at the Sprint level.

Additional information about Features and Capabilities can be found here: https://www.scaledagileframework.com/features-and-capabilities/

Roles

Product Manager - The Product Manager is responsible for the Program backlog and roadmap, which is the definitive repository required to meet the upcoming release objectives, support strategic initiatives at the Program level, and provide the overall product vision and roadmap. The Product Manager works with Product Owners to ensure that Features being developed deliver the value and meet the needs of customers and stakeholders based on the overall PI objectives.

Release Train Engineer (RTE) - The RTE is a Scrum Master of Scrum Masters, acting at the Program level to coordinate resources, actions, planning, execution against commitments, removal of impediments, and ultimately results of multiple Scrum Teams at the Program level.

Solution Train Engineer (STE) - The STE is a Scrum Master of RTEs, acting at the Portfolio level.

Product Manager/Director - Senior product leader who understands the big picture and manages multiple Product Owners.

Owns and manages the delivery Roadmap on a timescale of 2 or more Quarters ahead.
Owns leadership sponsorship when newfound knowledge requires a pivot of scope or initiative, and manages the governance required.
Engages senior leadership to establish and hold to commitments made.
Communicates upstream, while maintaining situational awareness of tactical work in flight.

Solution Architect - Operates at senior management or executive level - ensures that the Product Manager(s) remain aligned with strategic objectives and empowers them to fulfill commitments.

Owns the Strategic Roadmap, commitments, and delivery at the Portfolio level.
Maintains situational awareness of Solution Trains in flight.
Resolves organizational blocks - works with the Enterprise to surface and resolve impediments before they block workflow.

Enterprise Architect - The Enterprise Architect is responsible for the optimization of manual and automated processes into an integrated environment. They act as lead architects by directing, integrating, and facilitating the work of Solution Architects while working on the target and transition architecture.

Executive Sponsor - Leads or sponsors Portfolio-level planning, sets priorities and objectives at the Enterprise and/or Portfolio level. Confirms scope, priority, and value delivery objectives and supports teams to deliver on them.

Pros and Cons

Pros

Well-led and capable organizations can deliver “market-wide” and “market-making” disruptive capability at rapid pace, and in a manner that can be sustained and iteratively enhanced by the enterprise.
Establish an “Enterprise-wide” culture of sustainable achievement, technical excellence, and continuous improvement.
Leverage institutional deep knowledge and human capital to rapidly introduce innovative and competitive capabilities, and keep the best assets of the enterprise engaged and delivering great value over a long period of time.
Provides a framework for removing non-performers at the enterprise level.
Provides a framework for promoting and enabling high performers at the enterprise level.

Cons

Requires a VERY high level of leadership, organizational communication, and collaboration capability - generally across multiple “traditional” silos.
PI Planning is a resource-intensive exercise that requires Executive-level commitment, communication/coordination, and follow-through. It requires not only a great effort for all teams, but it also requires executives to “get their hands dirty”, which tends to be one of the biggest challenges.
All of the requirements of the Sprinting framework, PLUS:
- Executive-level understanding and leadership of Agile practices at a “hands-on” level. Talking about this topic is usually insufficient - Executive Champions will need to get involved with their vision and support, and live up to the responsibilities they ask of their teams.
- Leadership-level accountability, honesty, and open communications among Enterprise peers is REQUIRED. (This is a big ask for traditional leadership who are focused on controlling their own silos.)
- Enterprise-level ability to commit and dedicate the needed resources without behaving as a “Helicopter Manager”.

Entry Criteria

The entry criteria for an organization to adopt and implement Program Increment Planning Horizon is high. The organization should have multiple teams with experience and some level of maturity operating Scrum-based Agile development. Organizations should have strong competence with the Agile roles in addition to a clear Executive Vision for what is to be delivered, including:

Fully engaged Executive Product Leadership who is accountable for the performance and roadmap of the entire Product organization
Teams with the ability to operate Scrum (2-6 teams with similar performance and capabilities should be required)
ScrumMasters should be capable, fully committed, and engaged in their roles
Product Owners should be capable, fully committed, and engaged in their roles
Dev Teams must be capable, committed, and engaged in their roles

Team MUST be committed and focused on the PI objectives they’re responsible for. Teams that are “part time” or responsible for a variety of “Non-PI” deliverables struggle to perform in the PI Planning Horizon. Stakeholders must be engaged and committed to the teams they support. If stakeholders are not engaged and committed, the team will question the value of the work they’ve committed to. Stakeholders should be accountable to the Agile Delivery Process, and they should be available and engaged as needed based on team requirements.

Thank you for reading this series!

-Matt

Nautobot ChatOps with Cisco Meraki

2021-10-14T00:00:00+00:00

Network to Code is releasing a new Nautobot app—a plugin to interact with Cisco Meraki using the existing Nautobot ChatOps framework! This app comes with prepackaged commands to gather data about your Meraki environment via chat commands. The Nautobot ChatOps app lowers the barrier of entry by providing interactions with chat platforms of Mattermost, Microsoft Teams, Slack, and Webex Teams. The amount of code needed to generate ChatOps commands is low, and the Meraki ChatOps app can be expanded to include new commands to fit any number of use cases.

The Nautobot ChatOps Meraki app extends the capabilities of the Nautobot ChatOps app to include a new chat command. This is done by registering to the Python entry point in Nautobot plugin ChatOps that provides functionality to the code written to interact with Meraki.

Visit Nautobot Chatops with Meraki Repository for more details.

This app introduces the following subcommands to the meraki command:

get-organizations
get-admins
get-devices
get-networks
get-switchports
get-switchports-status
get-firewall-performance
get-wlan-ssids Query Meraki for all SSIDs for a given Network.
get-camera-recent Query Meraki Recent Camera Analytics.
get-clients Query Meraki for List of Clients.
get-neighbors Query Meraki for List of LLDP or CDP Neighbors.
configure-basic-access-port Configure an access port with description, VLAN, and state.
cycle-port Cycles a port on a given switch.

Get Organizations

Gather all the Meraki Organizations based on the API Key used during setup.

Get Admins

Based on an Organization Name, return the Meraki Admins.

Get Devices

Gathers devices from Meraki. Provides a device type option in order to limit scope.

Get Networks

Gathers names of networks from Meraki.

Get Switchports

Gathers switch ports configuration details from an MS(Meraki Switch Model) switch device.

Get Switchports Status

Gathers switch ports operating status from an MS(Meraki Switch Model) switch device.

Get Firewall Performance

Query Meraki with a firewall to device performance. This provides an integer value.

Get WLAN SSIDS

Query Meraki for all SSIDs for a given Network.

Get Camera Recent

Query Meraki Recent Camera Analytics.

Get Clients

Query Meraki for List of Clients.

Get Neighbors

Query Meraki for List of LLDP or CDP Neighbors based on a device.

Configure Basic Access Port

Configure an access port with description, VLAN, and state.

Cycle Port

Cycles a port on a given switch. Equivalent to a shutdown, no shutdown procedure.

Extending the App

Cisco Meraki is a cloud-managed infrastructure manager that includes multiple different products. Since Meraki can manage switches, security devices, cameras, and wireless, the initial plugin aims to provide some value in each of these product lines. Extending this plugin to include more commands is simple, and I encourage contributions back into this project.

More Apps Are on the Way!

This is just the start of what is possible in extending the Nautobot ChatOps ecosystem. Whether you want to write your own, or use one of the additional plugins that has been created by Network to Code, the ecosystem for ChatOps is going to continue to grow! Keep an eye out for additional ChatOps plugins to be announced here!

-Jeff

Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 3

2021-10-12T00:00:00+00:00

Following from Parts 1 and 2 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the Sprinting Planning Horizon.

Overview – Sprinting Planning Horizon

Sprinting is an Agile Planning Horizon used by Scrum teams to rapidly and iteratively deliver products of the highest possible value within timeboxes called Sprints. Sprinting is a fixed cadence of iterative cycles (typically two weeks but not shorter than 1 week and not more than 4 weeks). The short delivery cycles and collaborative nature of Sprinting makes this framework ideal for projects with rapidly changing, highly emergent requirements.

Roles in the Sprinting Framework

There are three primary roles in the Sprinting / Scrum framework:

Product Owner (PO) – The primary job of the product owner is to be the customer representative (the voice of the customer), turning the vision into a workable product backlog for the team to execute and managing the return on investment/business value of the product or service under development. The product owner brings clarity to the vision of the product to be delivered and works with customers and stakeholders to build the list of requirements that makes up the product backlog. The PO owns the product backlog, sets priority of the work, defines requirements and acceptance criteria, and ultimately accepts or rejects the work delivered.
ScrumMaster (SM) – The role of the ScumMaster is multi-faceted. The SM works to build and maintain a high-performing team, manages the various scrum ceremonies, removes blocks and impediments, keeps the team focused and on track, and keeps a view of the big picture while looking for places where the team can make improvements. The SM also has a role of protecting the team’s collective health and protects the team’s “psychological security” to operate properly.
Scrum Team Members – The team members deliver on the product vision and build the functionality needed to implement features from the prioritized product backlog. The Scrum Team should range between 3 to 10 members, and the collective team will possess all of the skills and capabilities required to deliver on the product vision. The Scrum Team should be composed of “T-shaped” individuals - which is to say that, while they may specialize in a particular technical area, they have a broad range of technical capabilities. The Scrum Team owns grooming & refinement of the backlog to get stories to a “ready to work” state. The Scrum Team also owns the “Sprint Commitment” – which is the team’s commitment for what work will be delivered in a given Sprint. They own end-to-end development, testing, documentation, and delivery of valuable product functionality.

Ceremonies

In the Sprinting framework, teams conduct numerous ceremonies to plan work, deliver work, and inspect & adapt for continuous improvement.

Sprint Planning - During the planning meeting, the team reviews the stories, estimates against team capacity and priority, and commits to the delivery of this body of work within the Sprint timebox.
Daily Standup - During the Sprint, the team participates in a Daily Standup to review work done the day prior, coordinate the work to be delivered that day, and identify any impediments/blocks to be resolved on that day.
Backlog Grooming/Refinement - Also during the Sprint, the team holds a number of backlog grooming & refinement meetings to prioritize and estimate the work that will be needed for subsequent Sprints.
Retrospective - At the end of the Sprint, the team reviews the work delivered vs what they committed to in Sprint planning. They take stock of what went well and what did not. They also identify initiatives for continued improvement, and when possible, they create stories for the next Sprint to account for taking that improvement action.
Sprint Demo - At the end of the Sprint, the team demonstrates the functionality completed to stakeholders and takes feedback for additional iterative changes/improvements for future Sprints. These action items are then represented in the Product Backlog and prioritized for delivery in subsequent Sprints.

Artifacts

The work artifact for a Sprint is the Story. Generally, one or more stories make up Features which represent a unit of functionality that delivers considerable business value and fulfills a customer need. In the Sprint planning horizon, a team commits to and delivers stories which are needed by the Customer. Delivery of features can span more than one Sprint, whereas delivery of stories should be contained within the Sprint that the story was committed to.

Typically, the stories will have more robust requirements, acceptance criteria, and task lists than a Kanban issue. The Sprint stories will also include an estimated weight (in story points), and they will generally have some way to document the actual time spent for review and measurement in the Retrospective following the Sprint.

Stories on the sprint board will also generally identify which Feature/Epic they contribute to – they will identify the initiative or sprint goal that each story is aligned to.

The Sprint Board

Generally, the sprint board is similar to a Kanban board - the main difference is that it represents a specific timebox with predetermined start and end dates. There can be more “status” columns or labels for the stories on the board, including identification of blocked work, work that has upstream/downstream dependencies, or work that is a “stretch goal” for a sprint.

Pros and Cons

Pros

Here are the conditions which lend themselves to a Sprinting-based planning horizon. The Sprinting framework is ideal for:

Complicated or complex projects where the requirements or technology (or both) are uncertain; stakeholders may not be certain about what technology will be best for achieving the desired value, or what the final product will look like (but they know what they need).
The priority and requirements of the work to be delivered are understood and accepted at a general and broad-enough level that the work can be planned and delivered within 2 – 8 weeks. This is to say that the business is “certain enough” about the product needs & requirements that the scope can be “locked” and committed for delivery within the sprint “timebox”; further, the delivered capability will be useful and valuable upon delivery.
Situations where the Team needs to collaborate with customers/stakeholders to discover and identify the detailed needs/requirements, and where flexibility and negotiation can happen in the delivery process.

Source: [“The Scrum Field Guide” by Mitch Lacey]

Cons

Here are the conditions which DO NOT lend themselves to a Sprinting-based planning horizon:

The organization cannot commit DEDICATED resources to the effort for all three Scrum roles (PO, SM, Team).
Delivery requirements are unstable and change continuously.
Stakeholders and/or customers are not consistently available to engage, collaborate, or provide critical feedback. Stakeholders and/or customers are unable or unwilling to actively participate in the Sprint process.
Roles lacking authority – individuals are designated to conduct a Scrum role without the authority to make concrete decisions for the team or organization they represent.
The inability to contain or control “Helicopter Managers” – Managers who undermine the Sprint framework by asking individual team members for work or support which is not sanctioned within the Team’s Sprint Commitment – and the lack of authority on the part of the SM, PO or Team Member to stop this behavior.

Entry Criteria

The entry criteria for teams to adopt a Sprint-based agile planning horizon is significant. The organization MUST be able to identify the proper resources for the Product Owner, Scrum Master, and Scrum Team roles. The Team members must own and be accountable to operating within the framework for both themselves and their Teams.

Further, the organization MUST dedicate those resources to achievement of the required objectives.

The organization MUST commit to upholding the standards and duties of the roles and team commitment. This includes:

Preserve the sanctity of defined Scrum roles
Commitment of dedicated capacity of the Scrum Team – 80%+ of all time for each member
Commitment to uphold and exercise Scrum Ceremonies
- Sprint Planning
- Daily Standups
- Backlog Grooming / Refinement
- Sprint Retrospectives
- Sprint Reviews / Sprint Demonstrations
Teams must be able to plan work in concert with Product Ownership and commit to the work within each sprint
Stakeholders should have a vested interest in the work being delivered, and make themselves available for feedback and collaboration as needed by the Scrum team

Thank you! The final post in this series is a closer look at the Program Increment Agile Planning Horizon.

-Matt

Intro to Automation Part 2 - New Tools for a New Network

2021-10-12T00:00:00+00:00

In my previous blog post, I mentioned a few tools you can get started with when beginning your journey into Network Automation. Today, I hope to dive into them further and explore them and a few others in more detail. While there are many tools available for network automation, some are more common than others and will serve as a good foundation when getting started.

If you missed Part 1 of this Intro to Automation series, you can find it here.

Tools Overview

As mentioned in Part 1 of this series, making the transition from Network Engineer to Network Automation Engineer requires a shift in how you think of your network. But with this shift comes a different set of tools you’ll start to use, which will help you as you progress into this field.

There are so many tools out there for Software Developers, but we as Network Engineers tend to think differently and use different tools. It can be overwhelming figuring out which tools to start out with. In this post, I hope to go over some of the most popular ones used by Network Automation Engineers and explain them in a way that you not only understand, but that shows their real-world value to a Network Engineer.

Before I get into tools and software themselves, I need to put in a disclaimer. Just like with all things, there is no “one-size-fits-all” tool or solution that will always work in every situation. Some of these tools will work most of the time, and some will work only in specific instances. My goal is to outline the ones that are the most popular today with a wide range of community or commercial support and explain the pros and cons to each.

I put Git as the first tool in the list on purpose because I believe it is one of the most important tools a Network Engineer can learn to use. Even if you never write a single script or piece of automation, Git can still be very useful for a Network Engineer.

There are many non-automation related use cases, including:

Config backups
Script backups
Version controlling
Auditing

Note: Regarding auditing, it can be helpful internally or for external auditors, but may require some advanced setup to meet different auditing and compliance criteria.

Git can be used locally without ever needing to create an account with popular online services like GitHub or Bitbucket. As a new Network Automation Engineer, you can use it to create backups of scripts you write or device configurations. These can be used later to undo a mistake you made, or even answer the timeless question, “when was the last change made and what was it?”

Git is not GitHub, just like the Linux kernel is not Ubuntu.

Git is the most popular version control system available, and used by the majority of developers around the world. It is open-source, easy to use, and with a wide user base has a lot of community support available. While it is natively used with CLI commands, there are various GUI applications that can make learning and using Git easier when getting started. A brief list of some popular ones are:

Or if you use an IDE for writing scripts, they usually include Git support as well. I discuss IDEs more in a later section below.

PlayStation in a Git Article?

There are many features and benefits to Git, but seeing as Git is a VCS or Version Control System, the main one I use Git for is keeping track of different files and how they change over time. In other words, keeping track of all changes made to a set of files and folders.

Here’s a fun analogy for video gamers when learning about Git. When I was a kid, I was playing Tomb Raider II on the original PlayStation. I got in the habit of frequently saving and did it so often I could save the game without thinking about it. Well at one point, I fell off a cliff at the end of a level and went to reload my last save. But muscle memory kicked in and I accidentally saved the game instead of loading it! I had to restart the whole level over from the beginning! I learned a lesson that day, and it’s stuck with me the rest of my gaming life: always have multiple save files, and backup those files whenever possible.

Git can be thought of similarly but with multiple files or directories. Every time you make a “commit” in Git (save your progress), it’s like saving your game in a brand-new save file every single time. Did you spend hours working on a script, only to have it break and you have no idea why? You can easily revert back to any previous snapshot (commit) and start over or use it as a reference point. Yes, I’ve done exactly this before too many times to count!

Fun fact: Git was created by Linus Torvalds, the same person who created Linux!

If there is one programming language I would recommend to a Network Engineer getting into automation, it is without a doubt Python. According to the TIOBE Index, as of September 2021, Python is the 2nd most popular programming language in the world, and about to take over the #1 spot from C. In fact, I’d be surprised if you haven’t heard of Python’s benefits by now or even started learning it. If you haven’t, the best time to start is today.

One of the driving factors behind its popularity is its ease of use and learning curve, specifically for people without programming backgrounds. The benefits of this make it very easy to get into and start writing something useful pretty fast, and yet it can still be used to write complex automation.

With such popularity comes large community support. While learning Python, when you run into a question, chances are someone else has already posted the answer online, and it’s a quick Google search away.

Some of the cons can be realized after using Python for a few years. For example, there are other languages that are inherently faster with certain tasks, though they’re more complicated to learn. Additionally, there are certain software development “best practices” that are taken care of for you under the hood of Python, but need to be learned when using other languages.

In summary, from a Network Engineer just getting into scripting and automation to seasoned senior-level engineers, Python is perfect.

Fun fact: Did you know that some network hardware vendors have native Python support built right in to their devices? For example, if it’s installed and enabled, you can run the command python from privileged exec mode on a Cisco 9k switch and load a Python prompt!

Hello World!

In the world of programming languages, there’s a concept called the “Hello World” program. Essentially, it’s when someone who is learning a new programming language learns just enough to print out the phrase “Hello World!” to the screen. It’s seen as a good starting point while learning the basics and is actually really fun to do!

To demonstrate the simplicity of Python when compared to the other programming languages, here’s a “Hello World” program written in the other 2 languages at the top of the TIOBE Index I mentioned earlier, C and Java:

Python

print("Hello World!")

#include <stdio.h>
int main() {
   printf("Hello World!");
}

Java

class HelloWorld {
  static public void main( String args[] ) {
    System.out.println( "Hello World!" );
  }
}

Python is about as simple as you can get and makes it easy to get started! As I mentioned before, you may not need to understand what everything in the C or Java examples is right away since Python handles most of that behind the scenes, but you will eventually want to learn what they are and why they’re important.

Important: When starting with Python, make sure you’re using and learning Python 3, not Python 2.

Ansible is an open-source automation platform used for managing multiple devices easily. It was acquired by Red Hat in 2015, and it remains one of the most popular open-source automation tools for network automation engineers. Ansible can be used for relatively simple playbooks (scripts that you run) for a single switch, all the way up to complex fleet management systems for thousands of devices!

While there are other open-source tools available that are similar, Ansible is the best and most popular choice for managing network devices for a few reasons:

Agentless - Ansible connects directly to a network device, usually over SSH but can use other methods, and does not require an “agent”, or other piece of software, to already be pre-installed on the device. Installing an agent on a network device for management is not feasible, so this is where Ansible works well compared to similar tools like Chef or Puppet.
Inventories - Want to configure 100 switches without manually connecting to them one at a time? This is where Ansible really shines! Just provide it with a specially formatted inventory file, which includes a list of devices and a few other parameters, and Ansible will handle connecting to them all behind the scenes.
Modular - Similar to Python, with Ansible’s popularity comes a wide range of modules (plugins) you can use. Some are submitted by the open-source community, while others are officially supported third-party modules (e.g., Arista EOS).
Customization - If you need Ansible to do something unique to your environment, or there is a feature not yet created by the open-source community, you can write your own using…..Python! That’s right, Ansible runs off of Python and natively supports custom Python scripts to be imported into Ansible playbooks.
Commercial Support - Since Red Hat’s acquisition of Ansible in 2015, companies can now purchase commercial support for Ansible through Red Hat, or even through third-party companies like Network to Code.

IDEs and Text Editors

Earlier I mentioned a couple popular IDEs available that are free for personal use, however I want to explain them in more detail. An IDE or fancy text editor aren’t things you hear about much, but they’re SO important when working with automation.

As a network engineer, my text editor of choice was a generic notepad-style application, where I mostly used it to write out a switch config before configuring the device by copy/pasting the text into the CLI.

When you get into automation, you’ll be spending a lot more time with scripts, configs, settings files, etc. For this reason, I highly recommend you get a good IDE or text editor right away. The more you use it, the more familiar you’ll become with it. Eventually, you’ll never want to work without it!

One feature that is an absolute must have for whichever program you choose is syntax highlighting. While different programs will use different colors for syntax highlighting, they all essentially work the same. Instead of explaining what this is, look at the below images and ask yourself this question: which one is easier to read through?

IDEs

An Integrated Development Environment, or IDE for short, is an application that contains many common tools used for writing software (or even basic scripts) and is frequently used by software developers. There are many benefits to IDEs, and the popular ones have too many features to list.

Some of the downsides to them are also found with their strengths. They can be complex with many settings that make no sense when first starting out scripting. However in my opinion, their benefits strongly outweigh the negatives and I encourage you to try one out and give it some time before giving up on it right away. Don’t worry about every button or feature, and focus on the basics. As you become more familiar with them, you’ll start learning their features and other benefits more and more.

Two of the most popular ones available today used by Network Automation Engineers are VSCode by Microsoft and PyCharm by JetBrains. The syntax highlighting example above is from this free VSCode extension for Cisco IOS configs, though both support many other color variations and file formats.

Text Editors

There are many, many good text editors out there. Ask anyone who’s been in IT long enough, and they’ll not only have a favorite but a list of reasons why it’s the best. The real answer is there is no “best” text editor, only the one that works best for you. Most popular GUI-based text editors now offer some level of built-in syntax highlighting, but not all. If you’re uncomfortable with starting out with an IDE right away, or if you just want something better than Notepad to use in your day-to-day activities, I list three of the more popular ones available right now that are free to use:

APIs

While not necessarily a specific tool, APIs are more of a back-end technology. In fact, I’m sure you’ve heard before how great APIs are by now from someone you know. Before I wrote my very first script, I had IT friends telling me how great APIs were and how they used them and loved them. But when trying to explain to me what they are, I couldn’t understand why they were so good. I wrote this section in a way that hopefully explains APIs to network engineers that have a hard time grasping their usefulness, and in a way I wish they had been explained to me years ago.

An API allows one application or system to be able to interact with a completely different application or system in a structured, predefined manner with expected inputs and outputs on both sides. Simplified, it is a way for two programs to be able to talk to each other.

An analogy would be how people talk to each other using the English language. If two people can both speak English, then they are both able to understand what each word means, know how to talk to someone so they understand what was said, know what to expect as a response, and what that response means. One person’s response may even differ to the exact same request if it comes from someone they know (authenticated) vs. coming from someone they do not (unauthenticated).

Think of an API like this. Amy natively speaks Spanish (application 1) and Bob natively speaks French (application 2). They normally can’t understand each other, but if they both agree to speak to each other in their secondary language English (APIs), they can communicate in a limited but effective manner. In that analogy, an API is not a translator, but a predefined set of rules (English) for Amy and Bob to talk to each other.

To take it a step further, some types of APIs (like REST APIs) can require another application to be authenticated before it will listen to what it has to say (process the data). In the previous analogy, it is similar to how if Amy is friends with Bob (authenticated), they may respond to each other in one way. However if a complete stranger named Charley (unauthenticated) walked up to Amy and started saying the same thing Bob was saying, she may respond differently.

Note: There are multiple types of APIs, each with its own sets of rules, data formats, communication methods, etc.

Previously as a network engineer, I never really understood why I would need to use them. As a network automation engineer, I now use APIs for my automation scripts to be able to interact with network devices.

Traditionally, if I want to enable an interface on a Cisco switch, I have to connect to the CLI on the switch over SSH, and run these commands:

switch01# config t
switch01(config)# interface FastEthernet1/1
switch01(config-if)# no shutdown
switch01(config-if)# end
switch01# copy running-config startup-config

Simple right? Well, at least simple for humans to perform and understand. However when you start writing scripts to do this, you’ll find it’s a lot harder and very unreliable to do it this way. When I started writing scripts to configure switches, I would have my script connect over SSH and configure it in the exact same way as I would via CLI. While this worked, there are faster, more reliable, and easier ways of doing so.

A common scenario occurs when you try to configure a setting across multiple network devices with different OSs, or sometimes even different versions of the same OS. For example, if you look at the AAA configuration for Cisco IOS, compared to Cisco NX-OS, then compare it again to Cisco ASA, they are all different. As an engineer, I can manually adjust the commands in the CLI on the fly, but in my scripting, I have to account for each variation I might encounter.

Important: I also have to account for variations I am not aware of!

This is where APIs come in. Instead of worrying about variations in each OS or how each command is different, what if I could have my script configure it using the same method and know for certain it will get configured as expected? Or if it fails, can I have it tell me there’s an error without breaking anything? Using an API, you absolutely can!

In this example, you can use a network device or application’s built-in API to send it specific data. It will then receive the request, and since it already knows what to expect, it’s able to parse it out, perform any action requested, then return data back to your script in a pre-formatted and expected way. If you send it information in a way that it isn’t expecting, or are missing information that’s necessary, it will let you know as well!

Examples of data returned can be anything, including:

Was the job successful?
Command output
Configurations
Errors encountered
etc.

Summary

It’s absolutely amazing how many tools and applications you can use when automating your network. It’s even more amazing knowing that most of them are either open-source or offer some sort of free licensing agreement.

I encourage you to start with learning the basics of the tools I’ve mentioned. You don’t have to become an expert in any of them right away. I’ve been using Python for years, and I still learn new things about it every day from my peers here at Network to Code!

I also encourage you to give back to the open-source and network automation community as you progress in your career. Join us in Slack, and feel free to participate in discussions and ask for advice. It’s a Slack community run by Network Automation Engineers for anyone interested in automation, network automation, general networking, or even non-network-related IT systems.

Many resources are available online to learn these tools. While many of them are free and written by the community, Network to Code offers excellent training for those that learn better in a structured class environment. We cover topics such as Python, Ansible, and even general network automation concepts.

Thanks for reading, and stay tuned for Part 3 in this series! Happy automating!

Intro to Automation Series

Part 1 - Rethink How You Think

Part 2 - New Tools for a New Network

To Be Continued…

Stay tuned for Part 3 in this series. I will provide a link for it here once it is published.

Scheduling and Approving Jobs in Nautobot

2021-10-07T00:00:00+00:00

One of Nautobot’s most extensible features is Jobs, which allows users to execute custom Python scripts on demand from the Nautobot UI. These custom tasks can be related to Nautobot, but they do not have to be. These can be used to populate data in Nautobot, validate data in Nautobot, or general network automation tasks.

As Nautobot 1.2.0 approaches, we want to highlight some exciting new capabilities with the Jobs feature:

These two addtions extend Jobs’ capabilities and flexibility by allowing users to better define and control workflows. These features together also further extend Nautobot’s capabilities as a platform.

Job Scheduling

Job scheduling allows users to schedule jobs to be run in the future and/or repeated periodically on an hourly, daily, or weekly basis.

Scheduling via the UI

Jobs will be able to be scheduled from the Nautobot UI when the user creates the job:

Scheduling via the API

Jobs will also be able to be scheduled programmatically via the REST APIs. We’ll demonstrate this with an example.

The Scheduled Jobs screenshot below shows no scheduled jobs:

This Python script will schedule the DeviceConnectionsReport to run weekly starting October 24, 2021:

import requests
import json

from pprint import pprint

class_path = "jobs/local/device_data_validation"

job_class_name = "DeviceConnectionsReport"

url = f"https://192.168.18.2/api/extras/{class_path}/{job_class_name}/run/"


data = {
  "schedule": {
    "name": "Weekly in the future example",
    "interval": "weekly",
    "start_time": "2021-10-24T15:00Z"
  }
}

payload = json.dumps(data)
headers = {
  'Authorization': 'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn',
  'Content-Type': 'application/json'
}

response = requests.post(url, headers=headers, data=payload, verify=False)

pprint(response.json())

Once the script completes, we see the scheduled job:

Job Approval

This feature allows job scripts to be flagged to require approval; jobs with this flag will not be executed immediately. Instead, they will be put into a Jobs Approval Queue where any user other than the Job submitter can approve or deny the queued Job or approve it for dry-run. Once approved, the job will be run immediately unless it is scheduled to run at a later time.

Jobs can be set to require approval when the approval_required = True attribute is set in the job script’s Meta object:

Approval via UI

Jobs can be approved in the Nautobot UI on the Jobs Approval Queue page:

Approval via API

Jobs can also be approved programmatically via REST calls. This example script below approves a job with a specific job id:

import requests

from pprint import pprint

job_id = "ba1852e4-10fa-4fd9-bddb-2d0bacdd21d9"

action = "approve"  # Could also be "deny" or "dry-run"

url = f"https://192.168.18.2/api/extras/scheduled-jobs/{job_id}/{action}/"

payload={}
headers = {
  'Authorization': 'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn'
}

response = requests.post(url, headers=headers, data=payload, verify=False)

pprint(response.json())

Summary

Scheduling and approving jobs gives users better control of Job workflows initiated from Nautobot. These two features will be arriving in Nautobot 1.2.0. Stay tuned!

Happy coding!

-Tim Fiola

Database Version Control with Nautobot

2021-10-06T00:00:00+00:00

Our Nautobots, Rollback! post in late August introduced the community to Dolt databases and the workflow Dolt databases enable in Nautobot. Dolt is enabled in Nautobot via the Version Control app. This post will discuss the value that the Version Control app brings with its Git-like workflows and how those workflows fit into existing Nautobot capabilities to keep network automation data clean.

Data Hygiene

In a network automation context, clean data is correct (error-free) and properly formatted to be usable by the automation.

A process to achieve and maintain clean network automation data would have these general steps:

Have more than one set of eyes on the data and/or checks and balances prior to adding/removing/changing data
Get data into a consistent structure/format
Ensure the ability to remove bad data

As a practical matter, no single magic bullet exists to perform all these steps: clean data requires a multi-tiered approach. The Version Control app adds value because it enables Git-like workflows for changes to production data.

Version Control Workflows Enable Better Data Hygiene

Nautobot running a Dolt database brings Git-like workflows to Nautobot, allowing a more collaborative process to get changes to network automation data into production. This collaborative process includes explicit checkpoints for review, discussion, and approval of changes to production data.

The Nautobots, Rollback! post introduced the comparison below, contrasting a Dolt-enabled Git-like workflow for database changes to a non-Git-like workflow.

Dolt	Non-Dolt (PostgreSQL/MySQL)
Create new branch from Master	———–
Switch to new branch	———–
Make changes to data	Make changes to data (changes are immediately active)
Submit a pull request for review	———–
Review changes	———–
Refine changes based on PR feedback	———–
Deploy changes deliberately (Merge)	———–
Assess changes to production environment	Assess changes to production environment
If there are problems, immediately roll back to last good state	If there are problems, undo the changes manually or via external automation

Version Control’s Dolt database brings capabilities and workflows that were previously not available in database-backed solutions.

There are two main benefits provided by the Dolt-enabled workflow:

It creates an explicit checkpoint for data review prior to merging changes to the production data.
If there are problems with new production data, a user can quickly and easily roll back parts of the merge, or do a wholesale rollback of the entire merge.

The Git-like workflow results in cleaner production data and allows for troublesome data merges to quickly be undone.

Both the conversations around and reviews of pull requests help ensure clean data enters production.

Being able to quickly and easily revert data changes in part or wholesale reduces the time required to fix mistakes.

Programmatic Access

As is true with any automation-centric platform, the Version Control app has extensive REST API capabilities.

Branch-Specific Data

The Version Control app allows multiple branches to exist, and it will often be necessary to programmatically query a given branch for information. This app allows programmatic REST queries for data in specific branches via the dolt-branch keyword in the header.

Here is some simple example code that queries the cdg-edge-augment branch for all devices with an edge role in the cdg site:

import requests

from pprint import pprint

url = "https://internal-dolt-server/api/dcim/devices/?site=cdg&role=edge"

payload={}
headers = {
  'Authorization': 'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn',
  'dolt-branch': 'cdg-edge-augment'
}

response = requests.request("GET", url, headers=headers, data=payload)

pprint(response.json())

Here are the results (heavily edited for readability!):

{'count': 3,
 'next': None,
 'previous': None,
 'results': [{
          ---<snip>---
              'display': 'cdg-edge-01',
          ---<snip>---
              },
             {
          ---<snip>---
              'display': 'cdg-edge-02',
          ---<snip>---
              },
             {
          ---<snip>---
              'display': 'cdg-edge-03',
          ---<snip>---
              }

Here are the results of the same script, but with the 'dolt-branch': 'cdg-edge-augment' header info removed:

{'count': 2,
 'next': None,
 'previous': None,
 'results': [{'asset_tag': None,
          ---<snip>---
              'display': 'cdg-edge-01',
          ---<snip>---
              'virtual_chassis': None},
             {'asset_tag': None,
          ---<snip>---
              'display': 'cdg-edge-02',
          ---<snip>---
              'virtual_chassis': None}]}

The cdg-edge-augment branch has an additional device (cdg-edge-03) that the main branch does not.

Version Control-specific API Endpoints

The Version Control app also has specific API endpoints for querying the following:

Branches
Commits
Pull Requests
Pull Request Comments

Test Drive the Version Control App Youself!

This app is still very much in the development phase and is not officially supported (as of the writing of this blog), but you can still test-drive the current functionality in a dev/test environment. The public repo is available on GitHub. There, you will find instructions for setting up the dev/test environment.

Wrapping Up

Keeping network automation data clean is important because it affects how well the automation can implement desired network state. Nautobot’s innovative Version Control app provides another layer of data protection to Nautobot’s existing data-protection measures to keep your network automation data clean.

Thank you, and have an awesome day!

-Tim

One Parser to Rule All Your Circuit Maintenance Notifications

2021-10-05T00:00:00+00:00

A few months ago, Network to Code released two open source projects (more info) to contribute to solving a common problem in modern networks: understand when a circuit is going through a planned maintenance. On one side, the circuit-maintenance-parser, to parse notifications into a common data structure, and on the other side, the nautobot-circuit-maintenance plugin, that uses the parser library to automatically populate the data into Nautobot. Following months of development, we are happy to announce the release of version 2.0.0 of the parser, which comes with a lot of improvements based on existing customer deployments and now covers 19 different ISPs!

`circuit-maintenance-parser` Library

In the mentioned blog, we acknowledged that we were not the first ones trying to solve this issue. We decided to adopt the format proposed here (using the iCalendar format) as our gold standard.

Now, you could be wondering: why do we need a parser library when there is a well-defined format? The answer is well-known… being just a recommendation it is not fully adopted by all the Network Service Providers (NSPs). So there is still a need to parse arbitrary data formats in order to obtain something conforming to a standard Maintenance containing the following attributes:

provider: identifies the provider of the service that is the subject of the maintenance notification.
account: identifies an account associated with the service that is the subject of the maintenance notification.
maintenance_id: contains text that uniquely identifies the maintenance that is the subject of the notification.
circuits: list of circuits affected by the maintenance notification and their specific impact.
status: defines the overall status or confirmation for the maintenance.
start: timestamp that defines the start date of the maintenance in GMT.
end: timestamp that defines the end date of the maintenance in GMT.
stamp: timestamp that defines the update date of the maintenance in GMT.
organizer: defines the contact information included in the original notification.

Please, refer to the BCOP to more details about these attributes.

This library aims to fill the current gap between an ideal representation of a circuit maintenance notification and the current providers’ formats, enabling automation to be built around these notifications.

The first version of the circuit-maintenance-parser had a simple workflow that eventually became a blocker to solve more complex use-cases and this required a new middleware that could combine multiple data using custom logic to process composed notifications, and be able to accommodate future new use-cases. More details about the logic workflow is available in the library Readme.

Supported Providers

Obviously, one of the key success indicators of the library is how many Providers are supported, and thanks to multiple examples seen from early adopters, the supported providers list has increased to 19 providers and it’s growing quickly.

AquaComms
AWS
Cogent
Colt
EuNetworks
GTT
HGC
Lumen
Megaport
Momentum
NTT
PacketFabric
Seaborn
Sparkle
Telia
Telstra
Turkcell
Verizon
Zayo

Moreover, the gold format is supported by default with the GenericProvider, so any NSP that sends the notification with the iCalendar format is supported by default.

How to Use It?

The circuit_maintenance_parser library requires two things:

The notificationdata: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).
The provider identifier: used to select the required Provider. Each Provider contains the logic to process the notificationdata using associated parsers.

Python Library

First step is to define the Provider that we will use to parse the notifications. By default, the GenericProvider (used when no other provider type is defined) will support parsing of iCalendar notifications using the recommended format.

from circuit_maintenance_parser import init_provider

generic_provider = init_provider()

type(generic_provider)
<class 'circuit_maintenance_parser.provider.GenericProvider'>

However, usually some Providers don’t fully implement the standard. Or perhaps some information is missing, for example the organizer email. We also support custom defined Providers that can be used to tailor the data extraction based on the notifications your organization receives :

ntt_provider = init_provider("ntt")

type(ntt_provider)
<class 'circuit_maintenance_parser.provider.NTT'>

Once we have the Provider ready, we need to initialize the data to process. We call it NotificationData and can be initialized from a simple content and type or from more complex structures, such as an email.

from circuit_maintenance_parser import NotificationData

raw_data = b"""BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Maint Note//https://github.com/maint-notification//
BEGIN:VEVENT
SUMMARY:Maint Note Example
DTSTART;VALUE=DATE-TIME:20151010T080000Z
DTEND;VALUE=DATE-TIME:20151010T100000Z
DTSTAMP;VALUE=DATE-TIME:20151010T001000Z
UID:42
SEQUENCE:1
X-MAINTNOTE-PROVIDER:example.com
X-MAINTNOTE-ACCOUNT:137.035999173
X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415
X-MAINTNOTE-IMPACT:OUTAGE
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-service
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2
X-MAINTNOTE-STATUS:TENTATIVE
ORGANIZER;CN="Example NOC":mailto:noone@example.com
END:VEVENT
END:VCALENDAR
"""

data_to_process = NotificationData.init_from_raw("ical", raw_data)

type(data_to_process)
<class 'circuit_maintenance_parser.data.NotificationData'>

Finally, we retrieve the maintenances (it is a List because a notification can contain multiple maintenances) from the data calling the get_maintenances method from the Provider instance:

maintenances = generic_provider.get_maintenances(data_to_process)

print(maintenances[0].to_json())
{
"account": "137.035999173",
"circuits": [
{
"circuit_id": "acme-widgets-as-a-service",
"impact": "NO-IMPACT"
},
{
"circuit_id": "acme-widgets-as-a-service-2",
"impact": "OUTAGE"
}
],
"end": 1444471200,
"maintenance_id": "WorkOrder-31415",
"organizer": "mailto:noone@example.com",
"provider": "example.com",
"sequence": 1,
"stamp": 1444435800,
"start": 1444464000,
"status": "TENTATIVE",
"summary": "Maint Note Example",
"uid": "42"
}

Notice that, either with the GenericProvider or NTT provider, we get the same result from the same parsed data, because they are using exactly the same Processor and Parser. The only difference is that NTT Provider will provide some custom default values for NTT in case the notification doesn’t contain this data. In this case, the notification contains all the information, so the custom defaults for Provider are not used.

ntt_maintenances = ntt_provider.get_maintenances(data_to_process)
assert ntt_maintenances == maintenances

CLI

There is also a cli entrypoint circuit-maintenance-parser which offers easy access to the library using few arguments:

data-file: file storing the notification.
data-type: ical, html or email, depending on the data type.
provider-type: to choose the right Provider. If empty, the GenericProvider is used.

circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo
Circuit Maintenance Notification #0
{
  "account": "my_account",
  "circuits": [
    {
      "circuit_id": "/OGYX/000000/ /ZYO /",
      "impact": "OUTAGE"
    }
  ],
  "end": 1601035200,
  "maintenance_id": "TTN-00000000",
  "organizer": "mr@zayo.com",
  "provider": "zayo",
  "sequence": 1,
  "stamp": 1599436800,
  "start": 1601017200,
  "status": "CONFIRMED",
  "summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span",
  "uid": "0"
}

How to Extend the Library?

Even though the library aims to include support for as many providers as possible, it’s likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new Provider is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.

First step is creating a new file: circuit_maintenance_parser/parsers/abcde.py. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from circuit_maintenance_parser.parser. In the example, we only need to import Html and in the child class implement the methods required by the class, in this case parse_html() which will return a dict with all the data that this Parser can extract. In this case we have to helper methods, _parse_bs and _parse_tables that implement the logic to navigate the notification data.

from typing import Dict
import bs4  # type: ignore
from bs4.element import ResultSet  # type: ignore
from circuit_maintenance_parser.parser import Html

class HtmlParserABCDE1(Html):
    def parse_html(self, soup: ResultSet) -> Dict:
        data = {}
        self._parse_bs(soup.find_all("b"), data)
        self._parse_tables(soup.find_all("table"), data)
        return [data]

    def _parse_bs(self, btags: ResultSet, data: Dict):
      ...

    def _parse_tables(self, tables: ResultSet, data: Dict):
      ...

Next step is to create the new Provider by defining a new class in circuit_maintenance_parser/provider.py. This class that inherits from GenericProvider only needs to define two attributes:

_processors: is a list of Processor instances that uses several data Parsers. In this example, we don’t need to create a new custom Processor because the combined logic serves well (the most likely case), and we only need to use the new defined HtmlParserABCDE1 and also the generic EmailDateParser that extract the email date. Also notice that you could have multiple Processors with different Parsers in this list, supporting several formats.
_default_organizer: this is a default helper to fill the organizer attribute in the Maintenance if the information is not part of the original notification.
_include_filter: mapping of data_types to a list of regex expressions that if provided have to match to parse the notification. This feature removes noise from notifications that are received from the same provider, but that are not related to circuit maintenance notifications.
_exclude_filter: antagonist mapping to define via regex which are the notifications that must not parsed.

class ABCDE(GenericProvider):
    _processors: List[GenericProcessor] = [
        CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]),
    ]
    _default_organizer = "noc@abcde.com"
    _include_filter = {EMAIL_HEADER_SUBJECT: ["Scheduled Maintenance"]}

And expose the new Provider in circuit_maintenance_parser/__init__.py:

from .provider import (
    GenericProvider,
    ABCDE,
    ...
)

SUPPORTED_PROVIDERS = (
    GenericProvider,
    ABCDE,
    ...
)

Last, but not least, you should update the tests!

Test the new Parser in tests/unit/test_parsers.py
Test the new Provider logic in tests/unit/test_e2e.py

… adding the necessary data samples in tests/unit/data/abcde/.

What’s Next?

Give it a try!, as the community is growing more and more Providers are going to be added and you can benefit from all of them. Also, developing a new Provider or Parser is straightforward, and you can contribute to the library via Pull Requests or Issues, providing notifications samples to develop.

As showed in How to use it?, you can easily integrate it with any automation application only passing the notification data and selecting the Provider that should be used to parse it, and then do what you want with the structured output.

And don’t forget that you could get this integrated with Nautobot SoT using the Circuit Maintenance Plugin.

-Christian

Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 2

2021-10-04T00:00:00+00:00

Following from Part 1 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the “Flow” Planning Horizon.

Overview: Flow Planning Horizon

Flow is a “just in time” delivery framework where the work is prioritized and visualized on a Kanban board.

The main objective of Flow is THROUGHPUT – the goal is to complete the highest priority work in a continuous flow. This is accomplished by MINIMIZING the work in progress (WIP) so that the team/individual only works on one thing at a time until that work is DONE, then they pull the next highest priority item from the “to-do” stack. Sometimes, work gets blocked waiting for external input or approval – in this case, teams can use a “blocked” column and pull the next highest priority item, but blocked items must be the top priority of any work in flight and be worked as soon as it becomes un-gated.

Because work is continuous flow, it does not need to be estimated. Daily standups are not required but often prove to be beneficial – this type of Flow is often referred to as “Scrum-ban”. Retrospectives are not required, but in practice they should be held at some regular interval (e.g., monthly) to review the team’s performance, throughput, and quality, and to address common impediments that should be resolved within the organization.

Work is visualized on a Kanban board. The Kanban board uses separate columns to identify work to do next, work in progress, and work completed.

The “To Do” column is a prioritized list of work to be done, where team members pick from the top of the list when taking on new work.
The “In Progress” column is a list of the work “in-flight” and being delivered at any given moment.
- The “Done” column represents the list of work completed. In Flow, this can also be two columns – an “In Review” column is often used to show work that is completed but requires review and acceptance in order to be considered “Done”. In this case, there will be a “done” or “accepted” state where the work is catalogued for historical record and measurements of the flow rate / velocity.

The Kanban board - Example:

Source: [https://leanicontechnology.com/what-agile-methodology-is-best-for-your-development-teams-in-2020/]{.ul}

Key elements to visualize:

Criteria required for any work item to advance to the next workflow state. For example: Items in “to-do” need to meet some definition of “ready to work” (clear requirements, data inputs identified, etc.) Items in “in progress” need to meet some standard definition of done such as dev complete for all acceptance criteria/requirements, test complete, unit test or code coverage requirements complete & documented, etc.
WIP limits – according to the team standard set and agreed upon, work in flight must remain within this limit to ensure smooth and “ASAP” throughput

Pros and Cons

Pros – Here are the conditions which lend themselves to a Flow-based planning horizon

The work is routine or repetitive (i.e., the work is well understood, common, frequent)
Helpdesk or operational support work - teams dedicated to working defects arising from an established production process or product/application
Work that is not plannable, or doesn’t require a detailed/defined pre-planning process (ad hoc work)
Work that is simple enough to not require regular and holistic inspect & adapt cycles
Teams that are not mature in Agile
- Specialized roles are not required (product owner, scrum master, etc.). However, many organizations find it beneficial to have a role that:
  - Owns prioritization, completeness requirements, and removal of impediments
  - Monitors throughput and delivery results, facilitates continuous improvement
- Agile ceremonies are not necessarily required (but often beneficial) such as daily standups, backlog grooming, work planning and commitment, demonstrations of functionality
- Work doesn’t go through a grooming/refinement and estimation process
- There is no requirement for strict adherence to team size guidelines per Agile Scrum team best practices (typically 3 to 10 individuals)
- There is no requirement for truly cross-functional capabilities of the team; often Flow teams can be composed of people who are specialized in one or a few types of work; they don’t require a truly cross-functional skill set in the same way that an Agile Scrum team does.
- There is no time-boxed cadence (though this can be a double-edged sword and lead to bad habits and inefficient processes unless it is managed and worked with discipline)

Cons – Here are the conditions which DO NOT lend themselves to a Flow-based planning horizon

Work that is unfamiliar to the organization (“build new” versus “known process”)
Work that is complex; work that has upstream or downstream dependencies that must be coordinated and negotiated (i.e., work that needs to be planned for execution both within and outside of the team)
- This is especially true for work that is composed of features/epics and strategic initiatives where there are many functional parts that make up a larger whole of value to be delivered
Work that requires multi-discipline skillsets for delivery of work (e.g., UI work, middleware, database, architectural design, security & compliance scope, etc.)
Work that requires a high degree of stakeholder, vendor, or other external collaboration and partnership

Entry Criteria

The entry criteria for teams to adopt a Flow-based Agile Planning Horizon is minimal. You need to know who the individuals working this process are, and the best practice is to have a known and consistent daily capacity for delivering work. The reason for this is so that you can measure velocity, results, and quality, and manage for velocity/quality/productivity improvement over time.

With “Scrum-ban”, the team is more well-defined and fits into the accepted range of team size (3 – 10 people). This is because this team will deliver their work via flow/Kanban BUT they will have daily standups to coordinate and plan for the work in flight. In a “Scrum-ban” framework, the team should also participate in Retrospective ceremonies at some defined interval (ie, monthly). The “Scrum-ban” team may also conduct customer/stakeholder demonstrations as needed.

Thank you! The next post in this series is a closer look at the Sprinting Agile Planning Horizon.

-Matt

Nautobot Relationships Part 2

2021-09-30T00:00:00+00:00

In Part 1 we introduced Nautobot’s new custom defined Relationships feature. Potential use cases were identified, and the process to configure the relationships using the web user interface was outlined. Nautobot Relationships are supported by the REST API (Application Programming Interface) for standard CRUD (Create, Read, Update, Delete) operations and also by the GraphQL API for read-only operations. This post will outline how to leverage the API, for a use case described in Part 1, to programmatically interact with custom-defined relationships.

REST

For most systems, REST is the de facto API exposed to clients. It is an HTTP-based service that supports stateless client-server interaction for data retrieval of each model in the Nautobot database. For a more detailed overview, check out the Nautobot docs on REST.

HTTP has functions that align to the typical CRUD operations. Nautobot’s RESTful API uses these HTTP functions to interact with the objects in the database. The same operations supported on the GUI are supported on the API. To demonstrate, we’ll create the ‘Circuit IPAddress’ relationship using the API.

Create Relationship: IPAddress - Circuit

The first step when dealing with the API is to consult the endpoint documentation. On the Home page, select the API button in the bottom right corner.

The API button will redirect to <server_url>/api/docs/ which is the Nautobot API endpoint documentation powered by Swagger. To create a Relationship object a POST operation to the /extras/relationships/ endpoint is required. The docs outline that the data field for the POST method has a number of required fields, highlighted with a red asterisk.

To execute the HTTP POST, we’ll develop a HTTP client using the Python requests HTTP library, which supports REST and non-REST API communications. Alternatively, the Swagger documentation can be used to test REST API requests directly in the web browser.

When creating the Relationship, a valid slug field consisting of letters, numbers, underscores or hyphens is required to uniquely identify this new object. In the GUI, this field is automatically generated; but in the API it needs to be manually defined. All other fields should be as described in the previous post.

INFO: Nautobot release 1.2 will introduce auto-generated slug fields for the API.

server_url = "https://demo.nautobot.com"
url = f"{server_url}/api/extras/relationships/"

token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

payload = {
    "name": "Circuit IPAddress",
    "description": "Custom relationship between circuits and IP addresses",
    "slug": "circuit-ipaddress",
    "type": "one-to-many",
    "source_type": "circuits.circuit",
    "source_label": "IP Address",
    "destination_type": "ipam.ipaddress",
    "destination_label": "Circuit",
}
headers = {"Authorization": f"Token {token}"}

response = requests.request("POST", url, headers=headers, json=payload)
print(json.dumps(response.json(), indent=2))

If the object was created successfully, the HTTP return code of 201 is returned, along with the JSON definition of the object.

{
  "id": "2b13240d-cec0-4eb5-a640-ce8f9f47fd9c",
  "url": "http://demo.nautobot.com/api/extras/relationships/2b13240d-cec0-4eb5-a640-ce8f9f47fd9c/",
  "name": "Circuit IPAddress",
  "slug": "circuit-ipaddress",
  "description": "Custom relationship between circuits and IP addresses",
  "type": "one-to-many",
  "source_type": "circuits.circuit",
  "source_label": "IP Address",
  "source_hidden": false,
  "source_filter": null,
  "destination_type": "ipam.ipaddress",
  "destination_label": "Circuit",
  "destination_hidden": false,
  "destination_filter": null
}

Relationship Association

The Relationship Association is a way to use the new model Relationship. It links instances of the models used in the source_type and destination_type. It’s analogous to selecting the remote end of the relationship using a drop-down list on the web GUI.

To create the association using the REST API, the uuid of each instance in the association is configured in the source_id and destination_id. The relationship field can also be configured with a uuid or, as in the example, a slug field can be provided within a dictionary, causing Nautobot to do an object lookup.

The uuid values can be retrieved with a GET request to the specific model API endpoint. To simplify the example, we’ll use static object ID’s for each instance in the association. A simple loop will iterate over the two IP address objects to associate each instance with the circuit using the new circuit-ipaddress Relationship.

server_url = "https://demo.nautobot.com"
url = f"{server_url}/api/extras/relationship-associations/"

token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

circuit_id = "8b109292-047d-4140-b711-a9524ebb220c"
ip_addr1_id = "94a902a3-e063-44b6-889c-d3023b901aff"
ip_addr2_id = "8b195702-eca2-4ce3-8d48-68af8d9f5b65"

headers = {"Authorization": f"Token {token}"}

for ip_addr_id in [ip_addr1_id, ip_addr2_id]:
    payload = {
        "relationship": {
            "slug": "circuit-ipaddress",
        },
        "source_type": "circuits.circuit",
        "source_id": circuit_id,
        "destination_type": "ipam.ipaddress",
        "destination_id": ip_addr_id,
    }

    response = requests.request("POST", url, headers=headers, json=payload)

    print(json.dumps(response.json(), indent=2))

Once again, if successfully completed, the HTTP return code of 201 is returned, along with the JSON definition of the object(s).

{
  "id": "830166ef-93af-4eb8-ba56-2fe9e5d15bbe",
  "relationship": {
    "id": "00a17433-3f59-437a-a413-3f6e58e7d5f0",
    "url": "http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/",
    "name": "Circuit IPAddress",
    "slug": "circuit-ipaddress",
    "display": "Circuit IPAddress"
  },
  "source_type": "circuits.circuit",
  "source_id": "8b109292-047d-4140-b711-a9524ebb220c",
  "destination_type": "ipam.ipaddress",
  "destination_id": "94a902a3-e063-44b6-889c-d3023b901aff"
}
{
  "id": "d3d53984-61d5-414a-adaa-eec352baddad",
  "relationship": {
    "id": "00a17433-3f59-437a-a413-3f6e58e7d5f0",
    "url": "http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/",
    "name": "Circuit IPAddress",
    "slug": "circuit-ipaddress",
    "display": "Circuit IPAddress"
  },
  "source_type": "circuits.circuit",
  "source_id": "8b109292-047d-4140-b711-a9524ebb220c",
  "destination_type": "ipam.ipaddress",
  "destination_id": "8b195702-eca2-4ce3-8d48-68af8d9f5b65"
}

Query Relationship Association

In the majority of the cases when using relationships, the client will have an existing object in context, e.g., a circuit. So the query will be from the perspective of that object. The uuid of the circuit can be used to query along with the Relationship slug to get back any relationship information. The circuit was the source_type of the relationship, so the circuit object uuid is provided as a value into the source_id query parameter.

server_url = "http://demo.nautobot.com"
token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

slug = "circuit-ipaddress"
circuit_id = "8b109292-047d-4140-b711-a9524ebb220c"
url = f"{server_url}/api/extras/relationship-associations/?relationship={slug}&source_id={circuit_id}"

headers = {"Authorization": f"Token {token}"}

response = requests.request("GET", url, headers=headers)

print(json.dumps(response.json(), indent=2))

The output return data count shows two objects, representing the two IP address objects associated with the circuit and their JSON data.

{
  "count": 2,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "830166ef-93af-4eb8-ba56-2fe9e5d15bbe",
      "relationship": {
        "id": "00a17433-3f59-437a-a413-3f6e58e7d5f0",
        "url": "http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/",
        "name": "Circuit IPAddress",
        "slug": "circuit-ipaddress",
        "display": "Circuit IPAddress"
      },
      "source_type": "circuits.circuit",
      "source_id": "8b109292-047d-4140-b711-a9524ebb220c",
      "destination_type": "ipam.ipaddress",
      "destination_id": "94a902a3-e063-44b6-889c-d3023b901aff"
    },
    {
      "id": "d3d53984-61d5-414a-adaa-eec352baddad",
      "relationship": {
        "id": "00a17433-3f59-437a-a413-3f6e58e7d5f0",
        "url": "http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/",
        "name": "Circuit IPAddress",
        "slug": "circuit-ipaddress",
        "display": "Circuit IPAddress"
      },
      "source_type": "circuits.circuit",
      "source_id": "8b109292-047d-4140-b711-a9524ebb220c",
      "destination_type": "ipam.ipaddress",
      "destination_id": "8b195702-eca2-4ce3-8d48-68af8d9f5b65"
    }
  ]
}

GraphQL

GraphQL is a recent technology, designed as a query language for APIs and a server-side application for resolving client queries. Refer to the Resources section, for some excellent material from my colleague Tim Fiola on GraphQL.

In Nautobot, GraphQL supports read-only operations. No updates or modifications can be executed through the GraphQL API. Its power is in complex object data retrieval and the supporting tools for the language, such as GraphiQL, which is an integrated client in Nautobot. GraphiQL can be used to develop queries based on the object model schema to retrieve structured data from the Nautobot GraphQL server. Select the GraphQL button on the bottom-right of the Nautobot Home page.

Query Relationship Association

Nautobot’s GraphQL implementation supports querying custom-defined relationships (and their associations) using the following format, with the hyphens replaced with underscores.

rel_<RELATIONSHIP_SLUG>

Building on the previous example of querying the circuit-ipaddress Relationship.

rel_circuit_ipaddress

In GraphQL a sample circuit query, filtered with a specific cid might look as follows.

Notice how the rel_circuit_ipaddress query object has changed the hyphens to underscores. Also note that attributes available to query within the relationship association depend on the type of object.

The sample query uses the interface, vrf and address fields to show how the Relationship can be used to extract complex association data for the related IP address object. All GraphQL object type fields can be traversed further to retrieve native or custom-defined relationships. This makes it very easy to get all the required data in just one GraphQL query.

An IP address is represented as an IPAddressType in the GraphQL schema. Schema documentation can be viewed using the Docs link on the left-hand side of the GraphQL page. The schema for IPAddressType displays all of the fields available to query.

Accessing the fields of the model within the relationship using GraphQL is in contrast to the REST API, which returns a reference to the instance. The client then has to make another REST API call to the object endpoint to get the related data.

In addition, the GraphQL server response contains only the fields requested in the query. Notice how this is different from a REST response that returns all fields for an endpoint and the client must filter to extract interested fields.

Conclusion

This series focused on how custom-defined relationships can provide a flexible approach to modeling networks using Nautobot as a source of truth. Use cases facilitated the overall concept; and the various different configuration methods highlight Nautobot’s versatility. Relationships are a key feature in the Nautobot system. If you have any specific use cases please let us know.

-Paddy

Resources

Getting Started with Django

2021-09-28T00:00:00+00:00

Django is a popular open-source Python-based framework enabling quick and easy development of web applications. This post will walk through the initial install and setup of Django in order to develop a simple web front end to run a number of network utilities found in Network to Code’s netutils library.

Django Setup

To get started it is necessary to install the Django python package using pip. As a best practice this example installs Django inside a Python virtual environment, which thus should be activated before running pip install. As shown below, Django has been successfully installed and verified by confirming the installed version of the module.

ntc@ubuntu:~$ source env/bin/activate
(env) ntc@ubuntu:~/django-projects$ python3 -m pip install django

(env) ntc@ubuntu:~/django-projects$ pip freeze | grep Django
Django==3.2.6

Create a Django Project

Once Django is installed, the first step to get up and running is to create a Django project. Using the django-admin startproject command we can create our first Django project, named network_utilities. In this instance the project has been created in a parent django-projects directory, which was created manually before starting the Django project and not created by Django itself.

(env) ntc@ubuntu:~/django-projects$ django-admin startproject network_utilities

Looking inside this directory we can see that Django automatically created a number of subdirectories and Python files. What has been created is a directory store of our main project settings.

(env) ntc@ubuntu:~/django-projects$ tree
.
└── network_utilities
    ├── manage.py
    └── network_utilities
        ├── asgi.py
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

2 directories, 6 files

Django creates a topmost directory with the same name as our project (i.e., network_utilities) which contains manage.py. This is the Python script most used to interact with Django for functions such as creating Django apps, starting the web server, updating database schema, etc.

In the project parent directory we have a subdirectory that also has the same name as our project. This directory contains a number of Python scripts used to manage and define project settings:

settings.py - Django project settings including declaring installed Django apps, database connections, etc.
urls.py - here we can define the urls for our project
asgi.py & wsgi.py - Web server environment settings

Django’s built-in web server is not suited for production use. When deploying to production you should consider a deployment platform such as wsgi. For the purposes of describing how to get started with Django, the built-in web server will suffice, thus the asgi.py and wsgi.py files will not be used.

Running Django for the First Time

We can verify the installation by running the Django server using the manage.py script. For this we should be in the parent network_utilities directory.

(env) ntc@ubuntu:~/django-projects/network_utilities$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
August 12, 2021 - 16:36:57
Django version 3.2.6, using settings 'network_utilities.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

As can be seen from the output, Django has launched on our localhost on TCP port 8000, i.e., http://127.0.0.1:8000. Browsing to this URL verifies Django has been installed correctly. To stop the server we simply send the CONTROL-C command as instructed above.

A Quick Note on Django Models

You may have noticed from the output above there is now a db.sqlite3 file included in the project root directory. This was created when we ran the Django runserver option with the manage.py script.

We were also warned when executing runserver that there are a number of unapplied migrations. One of the built-in features of Django is an Object-Relational Mapper (ORM) allowing you to define and manage database schemas without the need to understand SQL. Django uses models and migrations to manage your database schema.

Models are Python classes used to describe the database schema (i.e. your database table structure), stored in models.py under each Django application. Migrations are Python scripts that modify your database when changes to the models are made. In this post we are not creating our own models, but it is worth noting that Django by default supports a number of databases, including SQLite (the default, being used here), MySQL, and Oracle, and through third-party integrations can support others (configured in settings.py DATABASES dictionary).

There are a number of Django apps included by default (listed in settings.py under INSTALLED_APPS) which provide “under the hood” Django functionality. The unapplied migrations warning simply states that Django wants to create the necessary database schema for these apps. We can use the manage.py script to apply these initial migrations:

(env) ntc@ubuntu:~/django-projects/network_utilities$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK

Django Apps

Django uses the concept of an “application” to provide modularity to our code, one of the built-in advantages of using the framework. A Django app is simply a collection of code that enables a particular function for your web application. When creating the app, a number of Django components are “tied together” within the app, namely:

The view - contains code for the feature we wish to develop
The URL pattern - where we define URL’s and map to a view

The view is a Python function with an argument request which handles the requests for the web page we create. There are other optional components, but to get started, we will just define the view and URL within our Django app for our first simple webpage.

Again we use the manage.py script , this time to create our Django app. Here the app we create is called network_tools:

(env) ntc@ubuntu:~/django-projects/network_utilities$ python manage.py startapp network_tools

Django creates a network_tools directory and a number of files, including views.py which we will use to create our first view:

(env) ntc@ubuntu:~/django-projects/network_utilities$ tree
.
├── db.sqlite3
├── manage.py
├── network_tools
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── network_utilities
    ├── asgi.py
    ├── __init__.py
    ├── __pycache__
    │   ├── __init__.cpython-38.pyc
    │   ├── settings.cpython-38.pyc
    │   ├── urls.cpython-38.pyc
    │   └── wsgi.cpython-38.pyc
    ├── settings.py
    ├── urls.py
    └── wsgi.py

4 directories, 18 files

Views & URLs

In order to create our first view it is necessary to declare our Django app in the INSTALLED_APPS list in settings.py (in the network_utilities sub-directory).

network_utilities/settings.py

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'network_tools',
]

A simple home page can be created in the views.py file under the network_tools app. Here we create a simple function called home with a single argument, request. This argument will capture the HTTP requests which we will use later. The function returns a welcome message using Django’s HttpResponse, which we import on line 1.

network_tools/views.py

from django.http import HttpResponse

def home(request):
    return HttpResponse("Welcome to NTC netutils Django demo")

Next we must map a URL to this view. Currently we have one urls.py file in the project settings directory i.e. network_utilities/network_utilities. It is possible to declare all our URL’s here but we also have an option to create a per-Django-application urls.py. As well as modularity this approach allows us to have a common prefix for each app. This is the approach we will take allowing us to create a /netutils prefix as we add webpages. We will create this prefix later when adding pages for the netutils utilities, but first we will setup the project and application urls.py for the home page.

In the project urls.py we map the home page URL (signified via a blank path '' ) to our Django app by including network_tools.urls.

network_utilities/urls.py

from django.contrib import admin
from django.urls import path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('network_tools.urls'))
]

We then create a urls.py file in the application directory (network_utilities/network_tools) in order to map the URL to our view, you can see that the path ('') is the same in both files.

network_tools/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('', views.home, name='home'),
]

Now if we run the Django server again and browse to http://127.0.0.1:8000 we will be presented with the welcome message and we have created our first Django webpage.

Building the Web App

Having covered the basics of Django we can build our web frontend for the NTC netutils library (python -m pip install netutils)

So far we have created a simple webpage with a welcome message, now we can use Django templates to write HTML code to develop the page further. The templates support an inheritance model whereby we can ensure that each page we create has a similar look and feel. To start we will create a base.html file, from which all other pages will inherit.

First we must create a templates directory inside our app directory. Django by default will look in this directory for the base.html file. This directory should also have a sub-directory with the same name as the app (network_tools) where HTML files for all the app pages will be stored. Similarly a static directory with a network_tools sub directory is created to store an image that will be displayed on each page inherited from base.html.

(env) network_utilities/network_tools$ tree
├── static
│   └── network_tools
│       └── ntc-logo.png
├── templates
│   ├── base.html
│   └── network_tools
│       └── home.html

The base.html file includes 2 blocks (title and body) which act as placeholders for subsequent pages to display page specific content. The header also includes a PNG image (the Network to Code logo); each page that extends from base.html will display this image. In order to load the image the load static instruction is necessary i.e. line 1.

templates/base.html

{% load static %}
<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <title>{% block title %}{% endblock %}</title>
    <img src="{% static 'network_tools/ntc-logo.png' %}" width="200px">
</head>

<body>
    {% block body %}
    {% endblock %}
</body>

</html>

We can now update the home page by creating a home.html in the templates/network_tools directory. This HTML file extends from base.html and provides inputs to the title and body blocks.

templates/network_tools/home.html

{% extends "base.html" %}

{% block title %}Welcome{% endblock %}

{% block body %}
<h1> Welcome to NTC netutils Django front end demo</h1>

<body>
    <p>
        This is the webpage for running NTC netutils utilities.
    </p>
</body>
{% endblock %}

In order to utilise these HTML templates we must update our view. The Django render function is now used instead of directly constructing an HttpResponse, and thus we can remove the from django.http import HttpResponse statement.

network_tools/views.py

from django.shortcuts import render

def home(request):
    return render("network_tools/home.html")

We now have a home page which inherits the NTC logo from base.html.

Integrating netutils

NTC’s netutils Python library is a collection of tools to help with your network automation projects, installed using ‘pip’:

(env) ntc@ubuntu:~/django-projects$ python3 -m pip install netutils

For the purposes of this post we will create a frontend for 2 of the ‘netutils’ tools:

TCP Ping - verify if a host is listening on a particular TCP port
IP Address Utilities - e.g., netmask to CIDR notation and vice-versa

As with our home page, it is necessary to update a number of files in order to create each new page as follows:

ping.html - a new file in the templates/network_tools directory
views.py - define our view for each page
urls.py - In both the project and application directories allows us to map a URL to our new view

First we will create a link on our home page to our new page for the netutils tool. This is configured in home.html we created previously, we have added a sub header and HREF for the “Ping” page to specify a link destination.

templates/network_tools/home.html

{% extends "base.html" %}

{% block title %}Welcome{% endblock %}

{% block body %}
<h1> Welcome to NTC netutils Django front end demo.</h1>

<body>
    <p>
        This is the webpage for running NTC netutils utilities.
    </p>
    <h2> Supported Netutils Functions:</h2>
    <a href="{%url 'ping' %}">Ping</a>
</body>
{% endblock %}

The ping.html template will inherit from base.html as before. We can now complete the page specific details using the 2 blocks declared in base.html, title and body. The title simply labels our browser tab, with ‘TCP Ping’ in this instance. The body block allows us to code the input fields necessary for the netutils TCP Ping utility, note in the body we specify the HTTP method verb which we will use in the view. The netutils ping tool accepts 2 arguments (IP Address and TCP Port number) requiring a form field for each argument, both of which are given a name (“ip” and “number”) which can be referenced in a Django view. We also create a submit button to allow the user to execute the TCP Ping utility.

The HTML code includes input validation on the TCP port to ensure only an integer in the valid TCP port range is accepted. Also a test_result variable is displayed which will be declared in our Django view. A link back to the Home page is added at the bottom of the page.

templates/network_tools/ping.html

{% extends "base.html" %}

{% block title %}TCP Ping{% endblock %}

{% block body %}
<h2> TCP Ping </h2>
<p>
    Verifies whether a TCP port is open on a given IP address
</p>
<form action="/ping/" method="get">
    <label for="">Please enter Host IP Address:</label>
    <input name="ip">
    <div><label for="">Please enter TCP Port:</label>
        <input type="number" name="port" min="0" max="65535">
    </div>
    <div><button type="submit">Submit</button></div>
</form>
<div class="result">{{test_result|linebreaks}}</div>
<a href="{%url 'home' %}">home</a>
{% endblock %}

In views.py we declare a new function to define the code necessary to execute the ping tool. The required functions from the netutils library are imported, as shown on line 2 and 3. The request object contains the data entered by the user, i.e. the ‘ip’ and ‘port’ parameters specified in ping.html.

The “ping” function includes input validation to verify the user input. First it verifies an IP Address has been entered, in this instance we use the get method of the request.GET object to query the 'ip' key. If an IP Address has been entered the netutils is_ip method is used to ensure the IP Address is in a valid format. This input validation is in addition to verification of the TCP port number configured in the HTML template which is enforced by the browser rather than the views.py function.

Having validated the user input, the netutils tcp_ping function is called and the result stored in a test_result dictionary. The function return statement calls Django render to render the ping.html template and passes in the test_result dictionary.

network_tools/views.py

from django.shortcuts import render
from netutils.ping import tcp_ping
from netutils.ip import is_ip, cidr_to_netmask, netmask_to_cidr, is_netmask

def home(request):
    return render("network_tools/home.html")

def ping(request):
    test_result = {}
  
    if request.GET.get('ip') == '':
        test_result['test_result'] = f'Please enter an IP Address {request.GET}'
    elif 'ip' in request.GET:
        if is_ip(request.GET['ip']) is False:
            test_result['test_result'] = 'Please enter a valid IP Address'
            return render(request, "network_tools/ping.html", test_result)
        host_ip = request.GET['ip']
        host_port = request.GET['port']
        ping_result = tcp_ping(host_ip, host_port)
        if ping_result is True:
            test_result['test_result'] = f"Success: Port {host_port} is Open on host {host_ip}"
        else:
            test_result['test_result'] = f"Failure: Cannot open connection to Port {host_port} on host {host_ip}"
        
    return render(request, "network_tools/ping.html", test_result)

Finally we must update the project and application urls.py to map a URL to our django view.

In the project urls.py we can now declare a netutils prefix which will be prepended to pages for this Django app.

network_utilities/urls.py

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('netutils/', include('network_tools.urls'))
]

In the application urls.py we map the URL to the view.

network_tools/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('', views.home, name="home"),
    path('ping/', views.ping, name="ping"),
]

We now have a new webpage allowing the user to validate if an IP Address is listening on a particular TCP port. The input validation configured will ensure a message is displayed if an invalid IP Address or TCP Port number is entered. In the example output below we entered valid details and were returned a ‘Success’ message.

Adding Additional Pages

This completes all the necessary steps to enable a Django frontend on one of Network to Code’s netutils functions. In order to create additional pages (e.g. the IP Address utilities) we follow the same process:

Add a link to the new page on the home page in home.html.
Add an HTML template for our new page, e.g., ip_addresses.html.
Define the view in views.py, e.g., an ip_addr(request) function plus necessary code logic.
Update the application-specific urls.py to map a url to the new view.

At this stage most of the work is in views.py and the application specific urls.py. Below is shown the changes to these files to add the CIDR/Netmask conversion tool.

templates/network_tools/ip_address.html

{% block body %}
<h2> cidr_to_netmask</h2>
<p>
    Creates a decimal format of a CIDR value.
</p>
<form action="/ip_addr" method="get">
    <div><label for="">Please enter CIDR value:</label>
        <input type="number" name="cidr" min="1" max="32">
    </div>
    <div><button type="submit">Submit</button></div>
</form>
<div class="result">{{netmask_result|linebreaks}}</div>
</div>
<h2> netmask_to_cidr</h2>
<p>
    Creates a CIDR notation of a given subnet mask in decimal format.
</p>
<form action="/ip_addr" method="get">
    <div><label for="">Please enter a netmask value:</label>
        <input name="netmask">
    </div>
    <div><button type="submit">Submit</button></div>
</form>
<div class="result">{{cidr_result|linebreaks}}</div>
</div>
<a href="{%url 'home' %}">home</a>
{% endblock %}

network_tools/views.py

def ip_addr(request):
    test_result = {}
    
    if request.GET.get('cidr') == '':
        test_result['test_result'] = 'Please a valid CIDR value [1-24]'
    elif 'cidr' in request.GET:
        cidr = int(request.GET['cidr'])
        netmask_result = cidr_to_netmask(cidr)
        test_result['netmask_result'] = f"Netmask for CIDR value {cidr} is {netmask_result}"
    elif 'netmask' in request.GET:
        netmask = request.GET['netmask']
        if is_netmask(netmask) is False:
            test_result['cidr_result'] = 'Please enter a valid Netmask'
            return render(request, "network_tools/ip_address.html", test_result)
        cidr_value = netmask_to_cidr(netmask)
        test_result['cidr_result'] = f"CIDR Value for netmask {netmask} is {cidr_value}"

    return render(request, "network_tools/ip_address.html", test_result)

This gives us 2 additional tools to convert CIDR value to Netmask and vice-versa.

Summary

We have just scratched the surface of Django, but hopefully this will give you an understanding of how to quickly get started. I hope you find this blog post useful.

-Nicholas Davey

The NTC Mag

Agile Planning Horizons (And how to Select Them) - An Enterprise Guide - Part 4

Overview

Ceremonies

Artifacts

Roles

Pros and Cons

Pros

Cons

Entry Criteria

Nautobot ChatOps with Cisco Meraki

Get Organizations

Get Admins

Get Devices

Get Networks

Get Switchports

Get Switchports Status

Get Firewall Performance

Get WLAN SSIDS

Get Camera Recent

Get Clients

Get Neighbors

Configure Basic Access Port

Cycle Port

Extending the App

More Apps Are on the Way!

Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 3

Overview – Sprinting Planning Horizon

Roles in the Sprinting Framework

Ceremonies

Artifacts

The Sprint Board

Pros and Cons

Pros

Cons

Entry Criteria

Intro to Automation Part 2 - New Tools for a New Network

Tools Overview

PlayStation in a Git Article?

Hello World!

IDEs and Text Editors

IDEs

Text Editors

APIs

Summary

Intro to Automation Series

To Be Continued…

Scheduling and Approving Jobs in Nautobot

Job Scheduling

Scheduling via the UI

Scheduling via the API

Job Approval

Approval via UI

Approval via API

Summary

Database Version Control with Nautobot

Data Hygiene

Version Control Workflows Enable Better Data Hygiene

Programmatic Access

Branch-Specific Data

Version Control-specific API Endpoints

Test Drive the Version Control App Youself!

Wrapping Up

One Parser to Rule All Your Circuit Maintenance Notifications

circuit-maintenance-parser Library

Supported Providers

How to Use It?

Python Library

CLI

How to Extend the Library?

What’s Next?

Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 2

Overview: Flow Planning Horizon

The Kanban board - Example:

Pros and Cons

Entry Criteria

Nautobot Relationships Part 2

REST

Create Relationship: IPAddress - Circuit

Relationship Association

`circuit-maintenance-parser` Library