Jekyll2021-10-18T18:11:42+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comAgile Planning Horizons (And how to Select Them) - An Enterprise Guide - Part 42021-10-18T00:00:00+00:002021-10-18T00:00:00+00:00https://blog.networktocode.com/post/agile-planning-horizons-part-4<p>Following from Parts 2 and 3 of this series which provided an overview of the Flow and Sprinting planning horizons, this article provides a closer look at the Program Increment, a Scaled Agile Planning Horizon.</p> <h2 id="overview">Overview</h2> <p>Program Increment is a timebox artifact of SCALED AGILE operations. PI represents a committed Enterprise or Portfolio vision and execution plan. A PI is timebox for delivery of a Value Stream which is executed by “teams of teams” which are organized as Release Trains or Solution Trains.</p> <ul> <li><strong>Release Train</strong> - A long-lived team of Agile teams which delivers solutions in a value stream. The Release Train operates in synchronous cadence and practices regular, organized cross-communication around blocks and dependencies</li> <li><strong>Solution Train</strong> - An organized group of release trains that is focused on building large and complex solutions under a shared, over-arching business/technology mission.</li> </ul> <p><img src="../../../static/images/blog_posts/agile-planning-horizons-4/Agile-Trains-Image.png" alt="Table of Planning Horizons" /></p> <ul> <li>Source: <a href="https://d2o2utebsixu4k.cloudfront.net/media/images/1544077632343-A-Value-Stream-can-have-many-ARTs-within-it.jpg">https://d2o2utebsixu4k.cloudfront.net/media/images/1544077632343-A-Value-Stream-can-have-many-ARTs-within-it.jpg</a></li> </ul> <p>The Program Increment (PI) itself is a series of sprints (usually 4-6 sprints) in which multiple teams operate on a shared sprint cadence to deliver on one or more committed objectives of the PI.</p> <p>The PI framework is an Enterprise-level synchronization of Agile Teams to deliver on large-scale enterprise objectives. Naturally, this requires large-scale synchronization of cadence and planning ceremonies across all teams. This includes daily or weekly syncs as needed (i.e. Scrum of Scrums and/or PO sync). This isn’t a status report. Instead, it is about communicating and coordinating about blockers/impediments, dependencies and “sequence of event” needs in order to ensure results are accomplished within the PI timebox for ALL teams in the train.</p> <p>By synchronizing the PI planning and the following sprints across all Teams and Trains tasked with delivery, cross-domain planning is enabled. As a result, the Scaled Agile organization benefits from:</p> <ul> <li><strong>Structured Cadence</strong> <ul> <li>Improved predictability</li> <li>Improved stakeholder confidence in wait times for committed capability</li> <li>Support for regular planning and cross-functional coordination</li> <li>Limitation of batch sizes of work to discrete intervals</li> <li>Scheduled integration points</li> </ul> </li> <li><strong>Synchronized Cross-Domain Planning</strong> <ul> <li>Improved control of injection of new work</li> <li>Improved and routine dependency management</li> <li>Supports full-system integration and assessment</li> <li>Multiple feedback perspectives</li> <li>Facilitates the ability for all stakeholders to engage with each other at the same time</li> <li>Requirements and design planning occurs in a comprehensive way</li> <li>Important stakeholder decisions are accelerated</li> <li>Trains and Teams create and take responsibility for the PI plans created</li> <li>Sequences and dependencies are understood and accounted for among Trains and Teams</li> </ul> </li> </ul> <p>The PI Planning Horizon requires a focus on dependencies and sequence of execution across teams for the purpose of achieving aligned Enterprise goals. A more detailed overview of the Program Increment can be found here: [<a href="https://www.scaledagileframework.com/program-increment/">https://www.scaledagileframework.com/program-increment/</a>]</p> <h2 id="ceremonies">Ceremonies</h2> <p>There are a number of ceremonies which typically occur within the PI:</p> <p><strong>PI Planning</strong> - PI Planning is a Scaled Agile ceremony (usually lasting 1-3 days), in which multiple teams gather to establish, plan, and commit to the objectives to be delivered collectively by the teams. Each team will create and commit to its own objectives, but these will be in alignment with an over-arching vision and roadmap. The ceremony should have a standard agenda which starts with a presentation of the business context and vision, followed by team breakouts where each team’s plans and commitments are established and presented. A more detailed overview of this ceremony can be found here: <a href="https://www.scaledagileframework.com/pi-planning/">[https://www.scaledagileframework.com/pi-planning/]</a>]</p> <p><strong>Scrum of Scrums and PO Sync</strong> - Teams should have a built-in schedule to sync every 1 or 2 weeks (or more frequently if needed) to coordinate around various team or train dependencies and review progress and impediments to make plan adjustments as needed. This sync can occur as either a Scrum of Scrums or a PO Sync, or it can be combined into a single ceremony, as appropriate.</p> <p><strong>System Demo</strong> - The system demo is held collectively at the end of each sprint among all Teams in a Train. The objective is to demonstrate the current state of the solution overall, and gather feedback from sponsors/stakeholders/customers at the “Train Level”. This ceremony should be an integrated demonstration that provides an objective measure of functionality, progress, and ultimately value of the work delivered so far within the PI. This feedback provides an opportunity for the Train to validate its progress against PI objectives and/or to make course corrections. At the end of the last sprint in the PI, there is a “Final System Demo” which is delivered in the same way, and should be a fully integrated demonstration of the functional product achieved in the PI against the committed objectives. A more detailed review of this ceremony can be found here: [<a href="https://www.scaledagileframework.com/system-demo/">https://www.scaledagileframework.com/system-demo/</a>]</p> <p><strong>Inspect and Adapt Workshop</strong> - The Inspect and Adapt Workshop is held at the end of the PI. It should include all of the teams and stakeholders who participated in PI Planning and/or were involved in supporting the execution of the PI. This event allows the teams and stakeholders to inspect the end result of the PI and engage in collective exercises for the purpose of continuous learning and improvement. The workshop consists of three parts:</p> <ol> <li>Final System Demo</li> <li>Quantitative and Qualitative Assessment</li> <li>Retrospective and Problem-Solving Workshop</li> </ol> <p>A more detailed review of this ceremony can be found here: [<a href="https://www.scaledagileframework.com/inspect-and-adapt/">https://www.scaledagileframework.com/inspect-and-adapt/</a>]</p> <h2 id="artifacts">Artifacts</h2> <p><strong>Capability (or “Solution Capability”)</strong> - A Capability is a higher-level behavior or functionality that is identified, planned, refined, and committed for delivery by the Solution Train; typically the Solution Capability is delivered by multiple Release Trains. The Capability is composed of one or more Features and is sized to fit within the PI. Capabilities are usually created by Product Managers in collaboration with Product Owners and other key stakeholders. Capabilities are structured like features, but they are at a higher level of abstraction, describing system-level behavior and function in support of the definition and development of large solutions. Solution Capabilities comprise Strategic Solutions: the products, services and systems delivered to the customer.</p> <p><strong>Master Feature</strong> - In some Scaled Agile organizations or contexts, Master Features exist to help enterprises organize and group related Features. Master Features have the same properties as Features and Capabilities (overview, statement of benefit, and acceptance criteria) but exist at a level of abstraction higher than that of a Feature but lower than that of a Capability.</p> <p><strong>Feature</strong> - A Feature is a service or functionality that fulfills a stakeholder need.</p> <blockquote> <p><em>The Feature is the primary artifact, which is identified, estimated, and groomed to “ready state”, and ultimately planned and committed to by the Release Train within the sprints of a Program Increment.</em></p> </blockquote> <p>The feature should include a description of the service or function as well as a statement of benefit and acceptance criteria. The Feature is composed of one or more User Stories. Features are typically created by the Product Owner, with Solution (or System) Architects commonly creating enabler features that will be required for delivery of the functional Features and Capabilities. Features are committed to by Release Trains at the PI level, and the constituent Stories are committed to by Teams at the Sprint level.</p> <p>Additional information about Features and Capabilities can be found here: <a href="https://www.scaledagileframework.com/features-and-capabilities/">https://www.scaledagileframework.com/features-and-capabilities/</a></p> <h2 id="roles">Roles</h2> <p><strong>Product Manager</strong> - The Product Manager is responsible for the Program backlog and roadmap, which is the definitive repository required to meet the upcoming release objectives, support strategic initiatives at the Program level, and provide the overall product vision and roadmap. The Product Manager works with Product Owners to ensure that Features being developed deliver the value and meet the needs of customers and stakeholders based on the overall PI objectives.</p> <p><strong>Release Train Engineer (RTE)</strong> - The RTE is a Scrum Master of Scrum Masters, acting at the Program level to coordinate resources, actions, planning, execution against commitments, removal of impediments, and ultimately results of multiple Scrum Teams at the Program level.</p> <p><strong>Solution Train Engineer (STE)</strong> - The STE is a Scrum Master of RTEs, acting at the Portfolio level.</p> <p><strong>Product Manager/Director</strong> - Senior product leader who understands the big picture and manages multiple Product Owners.</p> <ul> <li>Owns and manages the delivery Roadmap on a timescale of 2 or more Quarters ahead.</li> <li>Owns leadership sponsorship when newfound knowledge requires a pivot of scope or initiative, and manages the governance required.</li> <li>Engages senior leadership to establish and hold to commitments made.</li> <li>Communicates upstream, while maintaining situational awareness of tactical work in flight.</li> </ul> <p><strong>Solution Architect</strong> - Operates at senior management or executive level - ensures that the Product Manager(s) remain aligned with strategic objectives and empowers them to fulfill commitments.</p> <ul> <li> <p>Owns the Strategic Roadmap, commitments, and delivery at the Portfolio level.</p> </li> <li>Maintains situational awareness of Solution Trains in flight.</li> <li>Resolves organizational blocks - works with the Enterprise to surface and resolve impediments before they block workflow.</li> </ul> <p><strong>Enterprise Architect</strong> - The Enterprise Architect is responsible for the optimization of manual and automated processes into an integrated environment. They act as lead architects by directing, integrating, and facilitating the work of Solution Architects while working on the target and transition architecture.</p> <p><strong>Executive Sponsor</strong> - Leads or sponsors Portfolio-level planning, sets priorities and objectives at the Enterprise and/or Portfolio level. Confirms scope, priority, and value delivery objectives and supports teams to deliver on them.</p> <h2 id="pros-and-cons">Pros and Cons</h2> <h3 id="pros">Pros</h3> <ul> <li>Well-led and capable organizations can deliver “market-wide” and “market-making” disruptive capability at rapid pace, and in a manner that can be sustained and iteratively enhanced by the enterprise.</li> <li>Establish an “Enterprise-wide” culture of sustainable achievement, technical excellence, and continuous improvement.</li> <li>Leverage institutional deep knowledge and human capital to rapidly introduce innovative and competitive capabilities, and keep the best assets of the enterprise engaged and delivering great value over a long period of time.</li> <li>Provides a framework for removing non-performers at the enterprise level.</li> <li>Provides a framework for promoting and enabling high performers at the enterprise level.</li> </ul> <h3 id="cons">Cons</h3> <ul> <li>Requires a VERY high level of leadership, organizational communication, and collaboration capability - generally across multiple “traditional” silos.</li> <li>PI Planning is a resource-intensive exercise that requires Executive-level commitment, communication/coordination, and follow-through. It requires not only a great effort for all teams, but it also requires executives to “get their hands dirty”, which tends to be one of the biggest challenges.</li> <li>All of the requirements of the Sprinting framework, PLUS: <ul> <li>Executive-level understanding and leadership of Agile practices at a “hands-on” level. Talking about this topic is usually insufficient - Executive Champions will need to get involved with their vision and support, and live up to the responsibilities they ask of their teams.</li> <li>Leadership-level accountability, honesty, and open communications among Enterprise peers is REQUIRED. (This is a big ask for traditional leadership who are focused on controlling their own silos.)</li> <li>Enterprise-level ability to commit and dedicate the needed resources without behaving as a “Helicopter Manager”.</li> </ul> </li> </ul> <h2 id="entry-criteria">Entry Criteria</h2> <p>The entry criteria for an organization to adopt and implement Program Increment Planning Horizon is <strong>high</strong>. The organization should have multiple teams with experience and some level of maturity operating Scrum-based Agile development. Organizations should have strong competence with the Agile roles in addition to a clear Executive Vision for what is to be delivered, including:</p> <ul> <li>Fully engaged Executive Product Leadership who is accountable for the performance and roadmap of the entire Product organization</li> <li>Teams with the ability to operate Scrum (2-6 teams with similar performance and capabilities should be required)</li> <li>ScrumMasters should be capable, fully committed, and engaged in their roles</li> <li>Product Owners should be capable, fully committed, and engaged in their roles</li> <li>Dev Teams must be capable, committed, and engaged in their roles</li> </ul> <p>Team MUST be committed and focused on the PI objectives they’re responsible for. Teams that are “part time” or responsible for a variety of “Non-PI” deliverables struggle to perform in the PI Planning Horizon. Stakeholders must be engaged and committed to the teams they support. If stakeholders are not engaged and committed, the team will question the value of the work they’ve committed to. Stakeholders should be accountable to the Agile Delivery Process, and they should be available and engaged as needed based on team requirements.</p> <p>Thank you for reading this series!</p> <p>-Matt</p>Matt RemkeFollowing from Parts 2 and 3 of this series which provided an overview of the Flow and Sprinting planning horizons, this article provides a closer look at the Program Increment, a Scaled Agile Planning Horizon.Nautobot ChatOps with Cisco Meraki2021-10-14T00:00:00+00:002021-10-14T00:00:00+00:00https://blog.networktocode.com/post/nautobot-chatops-with-meraki<p>Network to Code is releasing a new Nautobot app—a plugin to interact with Cisco Meraki using the existing Nautobot ChatOps framework! This app comes with prepackaged commands to gather data about your Meraki environment via chat commands. The Nautobot ChatOps app lowers the barrier of entry by providing interactions with chat platforms of Mattermost, Microsoft Teams, Slack, and Webex Teams. The amount of code needed to generate ChatOps commands is low, and the Meraki ChatOps app can be expanded to include new commands to fit any number of use cases.</p> <p>The Nautobot ChatOps Meraki app extends the capabilities of the Nautobot ChatOps app to include a new chat command. This is done by registering to the Python entry point in Nautobot plugin ChatOps that provides functionality to the code written to interact with Meraki.</p> <p>Visit <a href="https://github.com/nautobot/nautobot-plugin-chatops-meraki">Nautobot Chatops with Meraki Repository</a> for more details.</p> <p>This app introduces the following subcommands to the <code class="language-plaintext highlighter-rouge">meraki</code> command:</p> <ul> <li>get-organizations</li> <li>get-admins</li> <li>get-devices</li> <li>get-networks</li> <li>get-switchports</li> <li>get-switchports-status</li> <li>get-firewall-performance</li> <li>get-wlan-ssids Query Meraki for all SSIDs for a given Network.</li> <li>get-camera-recent Query Meraki Recent Camera Analytics.</li> <li>get-clients Query Meraki for List of Clients.</li> <li>get-neighbors Query Meraki for List of LLDP or CDP Neighbors.</li> <li>configure-basic-access-port Configure an access port with description, VLAN, and state.</li> <li>cycle-port Cycles a port on a given switch.</li> </ul> <h3 id="get-organizations">Get Organizations</h3> <p>Gather all the Meraki Organizations based on the API Key used during setup.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_meraki/00-meraki-get-orgs.png" alt="Meraki Get Organizations" /></p> <h3 id="get-admins">Get Admins</h3> <p>Based on an Organization Name, return the Meraki Admins.</p> <h3 id="get-devices">Get Devices</h3> <p>Gathers devices from Meraki. Provides a device type option in order to limit scope.</p> <h3 id="get-networks">Get Networks</h3> <p>Gathers names of networks from Meraki.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_meraki/00-meraki-get-networks.png" alt="Meraki Get Networks" /></p> <h3 id="get-switchports">Get Switchports</h3> <p>Gathers switch ports configuration details from an MS(Meraki Switch Model) switch device.</p> <h3 id="get-switchports-status">Get Switchports Status</h3> <p>Gathers switch ports operating status from an MS(Meraki Switch Model) switch device.</p> <h3 id="get-firewall-performance">Get Firewall Performance</h3> <p>Query Meraki with a firewall to device performance. This provides an integer value.</p> <h3 id="get-wlan-ssids">Get WLAN SSIDS</h3> <p>Query Meraki for all SSIDs for a given Network.</p> <h3 id="get-camera-recent">Get Camera Recent</h3> <p>Query Meraki Recent Camera Analytics.</p> <h3 id="get-clients">Get Clients</h3> <p>Query Meraki for List of Clients.</p> <h3 id="get-neighbors">Get Neighbors</h3> <p>Query Meraki for List of LLDP or CDP Neighbors based on a device.</p> <h3 id="configure-basic-access-port">Configure Basic Access Port</h3> <p>Configure an access port with description, VLAN, and state.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_meraki/00-meraki-port-config.png" alt="Meraki Configure Port 1" /> <img src="../../../static/images/blog_posts/nautobot_chatops_meraki/02-meraki-port-config.png" alt="Meraki Configure Port 2" /></p> <h3 id="cycle-port">Cycle Port</h3> <p>Cycles a port on a given switch. Equivalent to a shutdown, no shutdown procedure.</p> <h2 id="extending-the-app">Extending the App</h2> <p>Cisco Meraki is a cloud-managed infrastructure manager that includes multiple different products. Since Meraki can manage switches, security devices, cameras, and wireless, the initial plugin aims to provide some value in each of these product lines. Extending this plugin to include more commands is simple, and I encourage contributions back into this project.</p> <h2 id="more-apps-are-on-the-way">More Apps Are on the Way!</h2> <p>This is just the start of what is possible in extending the Nautobot ChatOps ecosystem. Whether you want to write your own, or use one of the additional plugins that has been created by Network to Code, the ecosystem for ChatOps is going to continue to grow! Keep an eye out for additional ChatOps plugins to be announced here!</p> <p>-Jeff</p>Jeff KalaNetwork to Code is releasing a new Nautobot app—a plugin to interact with Cisco Meraki using the existing Nautobot ChatOps framework! This app comes with prepackaged commands to gather data about your Meraki environment via chat commands. The Nautobot ChatOps app lowers the barrier of entry by providing interactions with chat platforms of Mattermost, Microsoft Teams, Slack, and Webex Teams. The amount of code needed to generate ChatOps commands is low, and the Meraki ChatOps app can be expanded to include new commands to fit any number of use cases.Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 32021-10-12T00:00:00+00:002021-10-12T00:00:00+00:00https://blog.networktocode.com/post/agile-planning-horizons-part-3<p>Following from Parts 1 and 2 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the Sprinting Planning Horizon.</p> <h2 id="overview--sprinting-planning-horizon">Overview – Sprinting Planning Horizon</h2> <p>Sprinting is an Agile Planning Horizon used by Scrum teams to rapidly and iteratively deliver products of the highest possible value within timeboxes called Sprints. Sprinting is a fixed cadence of iterative cycles (typically two weeks but not shorter than 1 week and not more than 4 weeks).  The short delivery cycles and collaborative nature of Sprinting makes this framework ideal for projects with rapidly changing, highly emergent requirements.</p> <h3 id="roles-in-the-sprinting-framework">Roles in the Sprinting Framework</h3> <p>There are three primary roles in the Sprinting / Scrum framework:</p> <ul> <li> <p><strong>Product Owner (PO)</strong> – The primary job of the product owner is to be the customer representative (the voice of the customer), turning the vision into a workable product backlog for the team to execute and managing the return on investment/business value of the product or service under development. The product owner brings clarity to the vision of the product to be delivered and works with customers and stakeholders to build the list of requirements that makes up the product backlog. The PO owns the product backlog, sets priority of the work, defines requirements and acceptance criteria, and ultimately accepts or rejects the work delivered.</p> </li> <li> <p><strong>ScrumMaster (SM)</strong> – The role of the ScumMaster is multi-faceted. The SM works to build and maintain a high-performing team, manages the various scrum ceremonies, removes blocks and impediments, keeps the team focused and on track, and keeps a view of the big picture while looking for places where the team can make improvements. The SM also has a role of protecting the team’s collective health and protects the team’s “psychological security” to operate properly.</p> </li> <li> <p><strong>Scrum Team Members</strong> – The team members deliver on the product vision and build the functionality needed to implement features from the prioritized product backlog. The Scrum Team should range between 3 to 10 members, and the collective team will possess all of the skills and capabilities required to deliver on the product vision. The Scrum Team should be composed of “T-shaped” individuals - which is to say that, while they may specialize in a particular technical area, they have a broad range of technical capabilities. The Scrum Team owns grooming &amp; refinement of the backlog to get stories to a “ready to work” state. The Scrum Team also owns the “Sprint Commitment” – which is the team’s commitment for what work will be delivered in a given Sprint. They own end-to-end development, testing, documentation, and delivery of valuable product functionality.</p> </li> </ul> <h3 id="ceremonies">Ceremonies</h3> <p>In the Sprinting framework, teams conduct numerous ceremonies to plan work, deliver work, and inspect &amp; adapt for continuous improvement.</p> <ul> <li> <p><strong>Sprint Planning</strong> - During the planning meeting, the team reviews the stories, estimates against team capacity and priority, and commits to the delivery of this body of work within the Sprint timebox. </p> </li> <li> <p><strong>Daily Standup</strong> - During the Sprint, the team participates in a Daily Standup to review work done the day prior, coordinate the work to be delivered that day, and identify any impediments/blocks to be resolved on that day. </p> </li> <li> <p><strong>Backlog Grooming/Refinement</strong> - Also during the Sprint, the team holds a number of backlog grooming &amp; refinement meetings to prioritize and estimate the work that will be needed for subsequent Sprints. </p> </li> <li> <p><strong>Retrospective</strong> - At the end of the Sprint, the team reviews the work delivered vs what they committed to in Sprint planning.  They take stock of what went well and what did not.  They also identify initiatives for continued improvement, and when possible, they create stories for the next Sprint to account for taking that improvement action.</p> </li> <li> <p><strong>Sprint Demo</strong> - At the end of the Sprint, the team demonstrates the functionality completed to stakeholders and takes feedback for additional iterative changes/improvements for future Sprints.  These action items are then represented in the Product Backlog and prioritized for delivery in subsequent Sprints.</p> </li> </ul> <h3 id="artifacts">Artifacts</h3> <p>The work artifact for a Sprint is the Story.  Generally, one or more stories make up Features which represent a unit of functionality that delivers considerable business value and fulfills a customer need.  In the Sprint planning horizon, a team commits to and delivers stories which are needed by the Customer.  Delivery of features can span more than one Sprint, whereas delivery of stories should be contained within the Sprint that the story was committed to.</p> <p>Typically, the stories will have more robust requirements, acceptance criteria, and task lists than a Kanban issue.  The Sprint stories will also include an estimated weight (in story points), and they will generally have some way to document the actual time spent for review and measurement in the Retrospective following the Sprint.</p> <p>Stories on the sprint board will also generally identify which Feature/Epic they contribute to – they will identify the initiative or sprint goal that each story is aligned to.</p> <h3 id="the-sprint-board">The Sprint Board</h3> <p>Generally, the sprint board is similar to a Kanban board - the main difference is that it represents a specific timebox with predetermined start and end dates.  There can be more “status” columns or labels for the stories on the board, including identification of blocked work, work that has upstream/downstream dependencies, or work that is a “stretch goal” for a sprint.</p> <h2 id="pros-and-cons">Pros and Cons</h2> <h3 id="pros">Pros</h3> <p>Here are the conditions which lend themselves to a Sprinting-based planning horizon. The Sprinting framework is ideal for:</p> <ul> <li> <p>Complicated or complex projects where the requirements or technology (or both) are uncertain; stakeholders may not be certain about what technology will be best for achieving the desired value, or what the final product will look like (but they know what they need).</p> </li> <li> <p>The priority and requirements of the work to be delivered are understood and accepted at a general and broad-enough level that the work can be planned and delivered within 2 – 8 weeks. This is to say that the business is “certain enough” about the product needs &amp; requirements that the scope can be “locked” and committed for delivery within the sprint “timebox”; further, the delivered capability will be useful and valuable upon delivery.</p> </li> <li> <p>Situations where the Team needs to collaborate with customers/stakeholders to discover and identify the detailed needs/requirements, and where flexibility and negotiation can happen in the delivery process.</p> </li> </ul> <p><img src="../../../static/images/blog_posts/agile-planning-horizons-3/Sprinting_Pros_Simple_to_Anarchy.png" alt="Sprinting Complicated to Complex" /></p> <p>Source: <a href="https://www.amazon.com/Scrum-Field-Guide-Addison-Wesley-Signature-dp-0133853624/dp/0133853624/ref=dp_ob_title_bk">[“The Scrum Field Guide” by Mitch Lacey]</a></p> <h3 id="cons">Cons</h3> <p>Here are the conditions which DO NOT lend themselves to a Sprinting-based planning horizon:</p> <ul> <li> <p>The organization cannot commit DEDICATED resources to the effort for all three Scrum roles (PO, SM, Team).</p> </li> <li> <p>Delivery requirements are unstable and change continuously.</p> </li> <li> <p>Stakeholders and/or customers are not consistently available to engage, collaborate, or provide critical feedback. Stakeholders and/or customers are unable or unwilling to actively participate in the Sprint process.</p> </li> <li> <p>Roles lacking authority – individuals are designated to conduct a Scrum role without the authority to make concrete decisions for the team or organization they represent.</p> </li> <li> <p>The inability to contain or control “Helicopter Managers” – Managers who undermine the Sprint framework by asking individual team members for work or support which is not sanctioned within the Team’s Sprint Commitment – and the lack of authority on the part of the SM, PO or Team Member to stop this behavior.</p> </li> </ul> <h2 id="entry-criteria">Entry Criteria</h2> <p>The entry criteria for teams to adopt a Sprint-based agile planning horizon is <strong>significant</strong>. The organization MUST be able to identify the proper resources for the Product Owner, Scrum Master, and Scrum Team roles. The Team members must own and be accountable to operating within the framework for both themselves and their Teams.</p> <p>Further, the organization MUST dedicate those resources to achievement of the required objectives.</p> <p>The organization MUST commit to upholding the standards and duties of the roles and team commitment. This includes:</p> <ul> <li> <p>Preserve the sanctity of defined Scrum roles</p> </li> <li> <p>Commitment of dedicated capacity of the Scrum Team – 80%+ of all time for each member</p> </li> <li> <p>Commitment to uphold and exercise Scrum Ceremonies</p> <ul> <li> <p>Sprint Planning</p> </li> <li> <p>Daily Standups</p> </li> <li> <p>Backlog Grooming / Refinement</p> </li> <li> <p>Sprint Retrospectives</p> </li> <li> <p>Sprint Reviews / Sprint Demonstrations</p> </li> </ul> </li> <li> <p>Teams must be able to plan work in concert with Product Ownership and commit to the work within each sprint</p> </li> <li> <p>Stakeholders should have a vested interest in the work being delivered, and make themselves available for feedback and collaboration as needed by the Scrum team</p> </li> </ul> <p>Thank you! The final post in this series is a closer look at the Program Increment Agile Planning Horizon.</p> <p>-Matt</p>Matt RemkeFollowing from Parts 1 and 2 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the Sprinting Planning Horizon.Intro to Automation Part 2 - New Tools for a New Network2021-10-12T00:00:00+00:002021-10-12T00:00:00+00:00https://blog.networktocode.com/post/new-tools-for-a-new-network<p>In my previous blog post, I mentioned a few tools you can get started with when beginning your journey into Network Automation. Today, I hope to dive into them further and explore them and a few others in more detail. While there are many tools available for network automation, some are more common than others and will serve as a good foundation when getting started.</p> <p>If you missed Part 1 of this Intro to Automation series, you can find it <a href="http://blog.networktocode.com/post/rethink-how-you-think/">here</a>.</p> <p> </p> <h2 id="tools-overview">Tools Overview</h2> <p>As mentioned in Part 1 of this series, making the transition from Network Engineer to Network Automation Engineer requires a shift in how you think of your network. But with this shift comes a different set of tools you’ll start to use, which will help you as you progress into this field.</p> <p>There are so many tools out there for Software Developers, but we as Network Engineers tend to think differently and use different tools. It can be overwhelming figuring out which tools to start out with. In this post, I hope to go over some of the most popular ones used by Network Automation Engineers and explain them in a way that you not only understand, but that shows their real-world value to a Network Engineer.</p> <p>Before I get into tools and software themselves, I need to put in a disclaimer. Just like with all things, there is no “one-size-fits-all” tool or solution that will always work in every situation. Some of these tools will work most of the time, and some will work only in specific instances. My goal is to outline the ones that are the most popular today with a wide range of community or commercial support and explain the pros and cons to each.</p> <p> </p> <h2><img src="../../../static/images/blog_posts/new_tools_for_a_new_network/git_logo.png" alt="Git" /></h2> <p>I put Git as the first tool in the list on purpose because I believe it is one of the most important tools a Network Engineer can learn to use. Even if you never write a single script or piece of automation, Git can still be very useful for a Network Engineer.</p> <p>There are many non-automation related use cases, including:</p> <ul> <li>Config backups</li> <li>Script backups</li> <li>Version controlling</li> <li>Auditing</li> </ul> <blockquote> <p><strong>Note:</strong> Regarding auditing, it can be helpful internally or for external auditors, but may require some advanced setup to meet different auditing and compliance criteria.</p> </blockquote> <p>Git can be used locally without ever needing to create an account with popular online services like GitHub or Bitbucket. As a new Network Automation Engineer, you can use it to create backups of scripts you write or device configurations. These can be used later to undo a mistake you made, or even answer the timeless question, “when was the last change made and what was it?”</p> <blockquote> <p>Git is not GitHub, just like the Linux kernel is not Ubuntu.</p> </blockquote> <p>Git is the most popular version control system available, and used by the majority of developers around the world. It is open-source, easy to use, and with a wide user base has a lot of community support available. While it is natively used with CLI commands, there are various GUI applications that can make learning and using Git easier when getting started. A brief list of some popular ones are:</p> <ul> <li><a href="https://www.sourcetreeapp.com/">Sourcetree</a> (my personal favorite)</li> <li><a href="https://www.sublimemerge.com/">Sublime Merge</a></li> <li><a href="https://www.gitkraken.com/">GitKraken</a></li> <li><a href="https://git-fork.com/">Fork</a></li> <li><a href="https://desktop.github.com/">GitHub Desktop</a></li> </ul> <p>Or if you use an IDE for writing scripts, they usually include Git support as well. I discuss IDEs more in a later section below.</p> <h3 id="playstation-in-a-git-article">PlayStation in a Git Article?</h3> <p>There are many features and benefits to Git, but seeing as Git is a <code class="language-plaintext highlighter-rouge">VCS</code> or <code class="language-plaintext highlighter-rouge">Version Control System</code>, the main one I use Git for is keeping track of different files and how they change over time. In other words, keeping track of <em>all</em> changes made to a set of files and folders.</p> <p>Here’s a fun analogy for video gamers when learning about Git. When I was a kid, I was playing Tomb Raider II on the original PlayStation. I got in the habit of frequently saving and did it so often I could save the game without thinking about it. Well at one point, I fell off a cliff at the end of a level and went to reload my last save. But muscle memory kicked in and I accidentally saved the game instead of loading it! I had to restart the whole level over from the beginning! I learned a lesson that day, and it’s stuck with me the rest of my gaming life: always have multiple save files, and backup those files whenever possible.</p> <p>Git can be thought of similarly but with multiple files or directories. Every time you make a “commit” in Git (save your progress), it’s like saving your game in a brand-new save file <em>every single time</em>. Did you spend hours working on a script, only to have it break and you have no idea why? You can easily revert back to any previous snapshot (commit) and start over or use it as a reference point. Yes, I’ve done exactly this before too many times to count!</p> <blockquote> <p><strong>Fun fact:</strong> Git was created by Linus Torvalds, the same person who created Linux!</p> </blockquote> <p> </p> <h2 id="-1"><img src="../../../static/images/blog_posts/new_tools_for_a_new_network/python_logo.png" alt="Python" /></h2> <p>If there is one programming language I would recommend to a Network Engineer getting into automation, it is without a doubt Python. According to the <a href="https://www.tiobe.com/tiobe-index/">TIOBE Index</a>, as of September 2021, Python is the 2nd most popular programming language in the world, and about to take over the #1 spot from C. In fact, I’d be surprised if you haven’t heard of Python’s benefits by now or even started learning it. If you haven’t, the best time to start is today.</p> <p>One of the driving factors behind its popularity is its ease of use and learning curve, specifically for people without programming backgrounds. The benefits of this make it very easy to get into and start writing something useful pretty fast, and yet it can still be used to write complex automation.</p> <p>With such popularity comes large community support. While learning Python, when you run into a question, chances are someone else has already posted the answer online, and it’s a quick Google search away.</p> <p>Some of the cons can be realized after using Python for a few years. For example, there are other languages that are inherently faster with certain tasks, though they’re more complicated to learn. Additionally, there are certain software development “best practices” that are taken care of for you under the hood of Python, but need to be learned when using other languages.</p> <p>In summary, from a Network Engineer just getting into scripting and automation to seasoned senior-level engineers, Python is perfect.</p> <blockquote> <p><strong>Fun fact:</strong> Did you know that some network hardware vendors have native Python support built right in to their devices? For example, if it’s installed and enabled, you can run the command <code class="language-plaintext highlighter-rouge">python</code> from privileged exec mode on a Cisco 9k switch and load a Python prompt!</p> </blockquote> <p> </p> <h3 id="hello-world">Hello World!</h3> <p>In the world of programming languages, there’s a concept called the “Hello World” program. Essentially, it’s when someone who is learning a new programming language learns just enough to print out the phrase “Hello World!” to the screen. It’s seen as a good starting point while learning the basics and is actually really fun to do!</p> <p>To demonstrate the simplicity of Python when compared to the other programming languages, here’s a “Hello World” program written in the other 2 languages at the top of the TIOBE Index I mentioned earlier, <code class="language-plaintext highlighter-rouge">C</code> and <code class="language-plaintext highlighter-rouge">Java</code>:</p> <p><strong>Python</strong></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">print</span><span class="p">(</span><span class="s">"Hello World!"</span><span class="p">)</span> </code></pre></div></div> <p><strong>C</strong></p> <div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdio.h&gt; </span><span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Hello World!"</span><span class="p">);</span> <span class="p">}</span> </code></pre></div></div> <p><strong>Java</strong></p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">HelloWorld</span> <span class="o">{</span> <span class="kd">static</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span> <span class="nc">String</span> <span class="n">args</span><span class="o">[]</span> <span class="o">)</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span> <span class="s">"Hello World!"</span> <span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <p>Python is about as simple as you can get and makes it easy to get started! As I mentioned before, you may not need to understand what everything in the <code class="language-plaintext highlighter-rouge">C</code> or <code class="language-plaintext highlighter-rouge">Java</code> examples is right away since Python handles most of that behind the scenes, but you will eventually want to learn what they are and why they’re important.</p> <blockquote> <p><strong>Important:</strong> When starting with Python, make sure you’re using and learning Python 3, not Python 2.</p> </blockquote> <p> </p> <h2 id="-2"><img src="../../../static/images/blog_posts/new_tools_for_a_new_network/ansible_logo.png" alt="Ansible" /></h2> <p>Ansible is an open-source automation platform used for managing multiple devices easily. It was acquired by Red Hat in 2015, and it remains one of the most popular open-source automation tools for network automation engineers. Ansible can be used for relatively simple playbooks (scripts that you run) for a single switch, all the way up to complex fleet management systems for thousands of devices!</p> <p>While there are other open-source tools available that are similar, Ansible is the best and most popular choice for managing network devices for a few reasons:</p> <ol> <li><strong>Agentless</strong> - Ansible connects directly to a network device, usually over SSH but can use other methods, and does not require an “agent”, or other piece of software, to already be pre-installed on the device. Installing an agent on a network device for management is not feasible, so this is where Ansible works well compared to similar tools like Chef or Puppet.</li> <li><strong>Inventories</strong> - Want to configure 100 switches without manually connecting to them one at a time? This is where Ansible really shines! Just provide it with a specially formatted inventory file, which includes a list of devices and a few other parameters, and Ansible will handle connecting to them all behind the scenes.</li> <li><strong>Modular</strong> - Similar to Python, with Ansible’s popularity comes a wide range of modules (plugins) you can use. Some are submitted by the open-source community, while others are officially supported third-party modules (e.g., <a href="https://docs.ansible.com/ansible/latest/collections/arista/eos/index.html">Arista EOS</a>).</li> <li><strong>Customization</strong> - If you need Ansible to do something unique to your environment, or there is a feature not yet created by the open-source community, you can write your own using…..Python! That’s right, Ansible runs off of Python and natively supports custom Python scripts to be imported into Ansible playbooks.</li> <li><strong>Commercial Support</strong> - Since Red Hat’s acquisition of Ansible in 2015, companies can now purchase commercial support for Ansible through Red Hat, or even through third-party companies like <a href="https://www.networktocode.com/services/support/">Network to Code</a>.</li> </ol> <p> </p> <h2 id="ides-and-text-editors">IDEs and Text Editors</h2> <p>Earlier I mentioned a couple popular IDEs available that are free for personal use, however I want to explain them in more detail. An IDE or fancy text editor aren’t things you hear about much, but they’re SO important when working with automation.</p> <p>As a network engineer, my text editor of choice was a generic notepad-style application, where I mostly used it to write out a switch config before configuring the device by copy/pasting the text into the CLI.</p> <p>When you get into automation, you’ll be spending a <em>lot</em> more time with scripts, configs, settings files, etc. For this reason, I highly recommend you get a good IDE or text editor right away. The more you use it, the more familiar you’ll become with it. Eventually, you’ll never want to work without it!</p> <p>One feature that is an absolute <em>must have</em> for whichever program you choose is syntax highlighting. While different programs will use different colors for syntax highlighting, they all essentially work the same. Instead of explaining what this is, look at the below images and ask yourself this question: which one is easier to read through?</p> <p><img src="../../../static/images/blog_posts/new_tools_for_a_new_network/no_syntax_highlighting.png" alt="No Syntax Highlighting" /></p> <p><img src="../../../static/images/blog_posts/new_tools_for_a_new_network/syntax_highlighting.png" alt="Syntax Highlighting" /></p> <h3 id="ides">IDEs</h3> <p>An Integrated Development Environment, or IDE for short, is an application that contains many common tools used for writing software (or even basic scripts) and is frequently used by software developers. There are many benefits to IDEs, and the popular ones have too many features to list.</p> <p>Some of the downsides to them are also found with their strengths. They can be complex with many settings that make no sense when first starting out scripting. However in my opinion, their benefits strongly outweigh the negatives and I encourage you to try one out and give it some time before giving up on it right away. Don’t worry about every button or feature, and focus on the basics. As you become more familiar with them, you’ll start learning their features and other benefits more and more.</p> <p>Two of the most popular ones available today used by Network Automation Engineers are <a href="https://code.visualstudio.com/">VSCode</a> by Microsoft and <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> by JetBrains. The syntax highlighting example above is from <a href="https://marketplace.visualstudio.com/items?itemName=jamiewoodio.cisco">this</a> free VSCode extension for Cisco IOS configs, though both support many other color variations and file formats.</p> <h3 id="text-editors">Text Editors</h3> <p>There are many, many good text editors out there. Ask anyone who’s been in IT long enough, and they’ll not only have a favorite but a list of reasons why it’s the best. The real answer is there is no “best” text editor, only the one that works best for you. Most popular GUI-based text editors now offer some level of built-in syntax highlighting, but not all. If you’re uncomfortable with starting out with an IDE right away, or if you just want something better than Notepad to use in your day-to-day activities, I list three of the more popular ones available right now that are free to use:</p> <ul> <li><a href="https://www.sublimetext.com/">Sublime</a></li> <li><a href="https://atom.io/">Atom</a></li> <li><a href="https://notepad-plus-plus.org/">Notepad++</a></li> </ul> <p> </p> <h2 id="apis">APIs</h2> <p>While not necessarily a specific tool, APIs are more of a back-end technology. In fact, I’m sure you’ve heard before how great APIs are by now from someone you know. Before I wrote my very first script, I had IT friends telling me how great APIs were and how they used them and loved them. But when trying to explain to me what they are, I couldn’t understand why they were so good. I wrote this section in a way that hopefully explains APIs to network engineers that have a hard time grasping their usefulness, and in a way I wish they had been explained to me years ago.</p> <p>An API allows one application or system to be able to interact with a completely different application or system in a structured, predefined manner with expected inputs and outputs on both sides. Simplified, it is a way for two programs to be able to talk to each other.</p> <p>An analogy would be how people talk to each other using the English language. If two people can both speak English, then they are both able to understand what each word means, know how to talk to someone so they understand what was said, know what to expect as a response, and what that response means. One person’s response may even differ to the exact same request if it comes from someone they know (authenticated) vs. coming from someone they do not (unauthenticated).</p> <p>Think of an API like this. Amy natively speaks Spanish (application 1) and Bob natively speaks French (application 2). They normally can’t understand each other, but if they both agree to speak to each other in their secondary language English (APIs), they can communicate in a limited but effective manner. In that analogy, an API is <em>not</em> a translator, but a predefined set of rules (English) for Amy and Bob to talk to each other.</p> <p>To take it a step further, some types of APIs (like REST APIs) can require another application to be authenticated before it will listen to what it has to say (process the data). In the previous analogy, it is similar to how if Amy is friends with Bob (authenticated), they may respond to each other in one way. However if a complete stranger named Charley (unauthenticated) walked up to Amy and started saying the same thing Bob was saying, she may respond differently.</p> <blockquote> <p><strong>Note:</strong> There are multiple types of APIs, each with its own sets of rules, data formats, communication methods, etc.</p> </blockquote> <p>Previously as a network engineer, I never really understood why I would need to use them. As a network automation engineer, I now use APIs for my automation scripts to be able to interact with network devices.</p> <p>Traditionally, if I want to enable an interface on a Cisco switch, I have to connect to the CLI on the switch over SSH, and run these commands:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>switch01# config t switch01(config)# interface FastEthernet1/1 switch01(config-if)# no shutdown switch01(config-if)# end switch01# copy running-config startup-config </code></pre></div></div> <p>Simple right? Well, at least simple for humans to perform and understand. However when you start writing scripts to do this, you’ll find it’s a lot harder and very unreliable to do it this way. When I started writing scripts to configure switches, I would have my script connect over SSH and configure it in the <em>exact same way</em> as I would via CLI. While this worked, there are faster, more reliable, and easier ways of doing so.</p> <p>A common scenario occurs when you try to configure a setting across multiple network devices with different OSs, or sometimes even different versions of the same OS. For example, if you look at the AAA configuration for Cisco IOS, compared to Cisco NX-OS, then compare it again to Cisco ASA, they are all different. As an engineer, I can manually adjust the commands in the CLI on the fly, but in my scripting, I have to account for each variation I might encounter.</p> <blockquote> <p><strong>Important:</strong> I <em>also</em> have to account for variations I am <strong><em>not</em></strong> aware of!</p> </blockquote> <p>This is where APIs come in. Instead of worrying about variations in each OS or how each command is different, what if I could have my script configure it using the same method <em>and</em> know for certain it will get configured as expected? Or if it fails, can I have it tell me there’s an error without breaking anything? Using an API, you absolutely can!</p> <p>In this example, you can use a network device or application’s built-in API to send it specific data. It will then receive the request, and since it already knows what to expect, it’s able to parse it out, perform any action requested, then return data back to your script in a pre-formatted and expected way. If you send it information in a way that it isn’t expecting, or are missing information that’s necessary, it will let you know as well!</p> <p>Examples of data returned can be anything, including:</p> <ul> <li>Was the job successful?</li> <li>Command output</li> <li>Configurations</li> <li>Errors encountered</li> <li>etc.</li> </ul> <h2 id="summary">Summary</h2> <p>It’s absolutely amazing how many tools and applications you can use when automating your network. It’s even more amazing knowing that most of them are either open-source or offer some sort of free licensing agreement.</p> <p>I encourage you to start with learning the basics of the tools I’ve mentioned. You don’t have to become an expert in any of them right away. I’ve been using Python for years, and I still learn new things about it every day from my peers here at <a href="https://www.networktocode.com/">Network to Code</a>!</p> <p>I also encourage you to give back to the open-source and network automation community as you progress in your career. Join us in <a href="https://slack.networktocode.com/">Slack</a>, and feel free to participate in discussions and ask for advice. It’s a Slack community run by Network Automation Engineers for anyone interested in automation, network automation, general networking, or even non-network-related IT systems.</p> <p>Many resources are available online to learn these tools. While many of them are free and written by the community, <a href="https://www.networktocode.com/">Network to Code</a> offers <a href="https://www.networktocode.com/training/">excellent training</a> for those that learn better in a structured class environment. We cover topics such as Python, Ansible, and even general network automation concepts.</p> <p>Thanks for reading, and stay tuned for Part 3 in this series! Happy automating!</p> <p> </p> <h2 id="intro-to-automation-series">Intro to Automation Series</h2> <p><a href="https://blog.networktocode.com/post/rethink-how-you-think/">Part 1 - Rethink How You Think</a></p> <p>Part 2 - New Tools for a New Network</p> <p> </p> <h2 id="to-be-continued">To Be Continued…</h2> <p>Stay tuned for Part 3 in this series. I will provide a link for it here once it is published.</p>Matt VitaleIn my previous blog post, I mentioned a few tools you can get started with when beginning your journey into Network Automation. Today, I hope to dive into them further and explore them and a few others in more detail. While there are many tools available for network automation, some are more common than others and will serve as a good foundation when getting started.Scheduling and Approving Jobs in Nautobot2021-10-07T00:00:00+00:002021-10-07T00:00:00+00:00https://blog.networktocode.com/post/scheduling-and-approving-jobs-in-nautobot<p>One of Nautobot’s most extensible features is <a href="https://nautobot.readthedocs.io/en/latest/additional-features/jobs/">Jobs</a>, which allows users to execute custom Python scripts on demand from the Nautobot UI. These custom tasks <strong>can</strong> be related to Nautobot, but they do not have to be. These can be used to populate data in Nautobot, validate data in Nautobot, or general network automation tasks.</p> <p>As Nautobot 1.2.0 approaches, we want to highlight some exciting new capabilities with the <em>Jobs</em> feature:</p> <ul> <li><a href="https://github.com/nautobot/nautobot/issues/125">Job Approval</a></li> <li><a href="https://github.com/nautobot/nautobot/issues/374">Job Scheduling</a></li> </ul> <p>These two addtions extend <em>Jobs’</em> capabilities and flexibility by allowing users to better define and control workflows. These features together also further extend Nautobot’s capabilities as a <em>platform</em>.</p> <h2 id="job-scheduling">Job Scheduling</h2> <p><em>Job scheduling</em> allows users to schedule jobs to be run in the future and/or repeated periodically on an hourly, daily, or weekly basis.</p> <h3 id="scheduling-via-the-ui">Scheduling via the UI</h3> <p>Jobs will be able to be scheduled from the Nautobot UI when the user creates the job:</p> <p><img src="../../../static/images/blog_posts/jobs_schedule_approval_teaser/1-job-schedule.png" alt="" /></p> <h3 id="scheduling-via-the-api">Scheduling via the API</h3> <p>Jobs will also be able to be scheduled programmatically via the REST APIs. We’ll demonstrate this with an example.</p> <p>The <em>Scheduled Jobs</em> screenshot below shows no scheduled jobs:</p> <p><img src="../../../static/images/blog_posts/jobs_schedule_approval_teaser/2-no-scheduled-jobs.png" alt="" /></p> <p>This Python script will schedule the <em>DeviceConnectionsReport</em> to run weekly starting October 24, 2021:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span> <span class="kn">import</span> <span class="nn">json</span> <span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span> <span class="n">class_path</span> <span class="o">=</span> <span class="s">"jobs/local/device_data_validation"</span> <span class="n">job_class_name</span> <span class="o">=</span> <span class="s">"DeviceConnectionsReport"</span> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"https://192.168.18.2/api/extras/</span><span class="si">{</span><span class="n">class_path</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">job_class_name</span><span class="si">}</span><span class="s">/run/"</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"schedule"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Weekly in the future example"</span><span class="p">,</span> <span class="s">"interval"</span><span class="p">:</span> <span class="s">"weekly"</span><span class="p">,</span> <span class="s">"start_time"</span><span class="p">:</span> <span class="s">"2021-10-24T15:00Z"</span> <span class="p">}</span> <span class="p">}</span> <span class="n">payload</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'Authorization'</span><span class="p">:</span> <span class="s">'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn'</span><span class="p">,</span> <span class="s">'Content-Type'</span><span class="p">:</span> <span class="s">'application/json'</span> <span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">())</span> </code></pre></div></div> <p>Once the script completes, we see the scheduled job:</p> <p><img src="../../../static/images/blog_posts/jobs_schedule_approval_teaser/3-scheduled-jobs.png" alt="" /></p> <h2 id="job-approval">Job Approval</h2> <p>This feature allows job scripts to be flagged to require approval; jobs with this flag will <strong>not</strong> be executed immediately. Instead, they will be put into a Jobs Approval Queue where any user <em>other than the Job submitter</em> can approve or deny the queued Job or approve it for <em>dry-run</em>. Once approved, the job will be run immediately unless it is scheduled to run at a later time.</p> <p>Jobs can be set to require approval when the <code class="language-plaintext highlighter-rouge">approval_required = True</code> attribute is set in the job script’s <code class="language-plaintext highlighter-rouge">Meta</code> object:</p> <p><img src="../../../static/images/blog_posts/jobs_schedule_approval_teaser/4-approval_required.png" alt="" /></p> <h3 id="approval-via-ui">Approval via UI</h3> <p>Jobs can be approved in the Nautobot UI on the <em>Jobs Approval Queue</em> page:</p> <p><img src="../../../static/images/blog_posts/jobs_schedule_approval_teaser/5-approval_queue.png" alt="" /></p> <h3 id="approval-via-api">Approval via API</h3> <p>Jobs can also be approved programmatically via REST calls. This example script below approves a job with a specific job id:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span> <span class="n">job_id</span> <span class="o">=</span> <span class="s">"ba1852e4-10fa-4fd9-bddb-2d0bacdd21d9"</span> <span class="n">action</span> <span class="o">=</span> <span class="s">"approve"</span> <span class="c1"># Could also be "deny" or "dry-run" </span> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"https://192.168.18.2/api/extras/scheduled-jobs/</span><span class="si">{</span><span class="n">job_id</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">action</span><span class="si">}</span><span class="s">/"</span> <span class="n">payload</span><span class="o">=</span><span class="p">{}</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'Authorization'</span><span class="p">:</span> <span class="s">'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn'</span> <span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">())</span> </code></pre></div></div> <h2 id="summary">Summary</h2> <p>Scheduling and approving jobs gives users better control of Job workflows initiated from Nautobot. These two features will be arriving in Nautobot 1.2.0. Stay tuned!</p> <p>Happy coding!</p> <p>-Tim Fiola</p>Tim FiolaOne of Nautobot’s most extensible features is Jobs, which allows users to execute custom Python scripts on demand from the Nautobot UI. These custom tasks can be related to Nautobot, but they do not have to be. These can be used to populate data in Nautobot, validate data in Nautobot, or general network automation tasks.Database Version Control with Nautobot2021-10-06T00:00:00+00:002021-10-06T00:00:00+00:00https://blog.networktocode.com/post/database-version-control-with-nautobot<p>Our <a href="https://blog.networktocode.com/post/nautobots-rollback/">Nautobots, Rollback!</a> post in late August introduced the community to Dolt databases and the workflow Dolt databases enable in Nautobot. Dolt is enabled in Nautobot via the <em>Version Control</em> app. This post will discuss the value that the <a href="https://github.com/nautobot/nautobot-plugin-version-control"><em>Version Control</em> app</a> brings with its Git-like workflows and how those workflows fit into existing Nautobot capabilities to keep network automation data <em>clean</em>.</p> <h2 id="data-hygiene">Data Hygiene</h2> <p>In a network automation context, <em>clean</em> data is correct (error-free) and properly formatted to be usable by the automation.</p> <p>A process to achieve and maintain clean network automation data would have these general steps:</p> <ul> <li>Have more than one set of eyes on the data and/or checks and balances prior to adding/removing/changing data</li> <li>Get data into a consistent structure/format</li> <li>Ensure the ability to remove bad data</li> </ul> <p>As a practical matter, no single magic bullet exists to perform all these steps: clean data requires a multi-tiered approach. The <em>Version Control</em> app adds value because it enables Git-like workflows for changes to production data.</p> <h2 id="version-control-workflows-enable-better-data-hygiene">Version Control Workflows Enable Better Data Hygiene</h2> <p>Nautobot running a Dolt database brings Git-like workflows to Nautobot, allowing a more collaborative process to get changes to network automation data into production. This collaborative process includes explicit checkpoints for review, discussion, and approval of changes to production data.</p> <p>The <a href="https://blog.networktocode.com/post/nautobots-rollback/">Nautobots, Rollback!</a> post introduced the comparison below, contrasting a Dolt-enabled Git-like workflow for database changes to a non-Git-like workflow.</p> <table> <thead> <tr> <th>Dolt</th> <th>Non-Dolt (PostgreSQL/MySQL)</th> </tr> </thead> <tbody> <tr> <td>Create new branch from Master</td> <td>———–</td> </tr> <tr> <td>Switch to new branch</td> <td>———–</td> </tr> <tr> <td>Make changes to data</td> <td>Make changes to data (changes are immediately active)</td> </tr> <tr> <td>Submit a pull request for review</td> <td>———–</td> </tr> <tr> <td>Review changes</td> <td>———–</td> </tr> <tr> <td>Refine changes based on PR feedback</td> <td>———–</td> </tr> <tr> <td>Deploy changes deliberately (Merge)</td> <td>———–</td> </tr> <tr> <td>Assess changes to production environment</td> <td>Assess changes to production environment</td> </tr> <tr> <td>If there are problems, immediately roll back to last good state</td> <td>If there are problems, undo the changes manually or via external automation</td> </tr> </tbody> </table> <p>Version Control’s Dolt database brings capabilities and workflows that were previously not available in database-backed solutions.</p> <p>There are two main benefits provided by the Dolt-enabled workflow:</p> <ol> <li>It creates an explicit checkpoint for data review prior to merging changes to the production data.</li> <li>If there are problems with new production data, a user can quickly and easily roll back parts of the merge, or do a wholesale rollback of the entire merge.</li> </ol> <p>The Git-like workflow results in cleaner production data and allows for troublesome data merges to quickly be undone.</p> <p><img src="../../../static/images/blog_posts/version-control-intro/pr-conversation.png" alt="" /> <em>Both the conversations around and reviews of pull requests help ensure clean data enters production.</em></p> <p><img src="../../../static/images/blog_posts/version-control-intro/revert-page.png" alt="" /> <em>Being able to quickly and easily revert data changes in part or wholesale reduces the time required to fix mistakes.</em></p> <h2 id="programmatic-access">Programmatic Access</h2> <p>As is true with any automation-centric platform, the <em>Version Control</em> app has extensive REST API capabilities.</p> <h3 id="branch-specific-data">Branch-Specific Data</h3> <p>The <em>Version Control</em> app allows multiple branches to exist, and it will often be necessary to programmatically query a given branch for information. This app allows programmatic REST queries for data in specific branches via the <code class="language-plaintext highlighter-rouge">dolt-branch</code> keyword in the header.</p> <p>Here is some simple example code that queries the <code class="language-plaintext highlighter-rouge">cdg-edge-augment</code> branch for all devices with an <code class="language-plaintext highlighter-rouge">edge</code> role in the <code class="language-plaintext highlighter-rouge">cdg</code> site:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span> <span class="n">url</span> <span class="o">=</span> <span class="s">"https://internal-dolt-server/api/dcim/devices/?site=cdg&amp;role=edge"</span> <span class="n">payload</span><span class="o">=</span><span class="p">{}</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'Authorization'</span><span class="p">:</span> <span class="s">'Token nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn'</span><span class="p">,</span> <span class="s">'dolt-branch'</span><span class="p">:</span> <span class="s">'cdg-edge-augment'</span> <span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">"GET"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span> <span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">())</span> </code></pre></div></div> <p>Here are the results (heavily edited for readability!):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{'count': 3, 'next': None, 'previous': None, 'results': [{ ---&lt;snip&gt;--- 'display': 'cdg-edge-01', ---&lt;snip&gt;--- }, { ---&lt;snip&gt;--- 'display': 'cdg-edge-02', ---&lt;snip&gt;--- }, { ---&lt;snip&gt;--- 'display': 'cdg-edge-03', ---&lt;snip&gt;--- } </code></pre></div></div> <p>Here are the results of the same script, but with the <code class="language-plaintext highlighter-rouge">'dolt-branch': 'cdg-edge-augment'</code> header info removed:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{'count': 2, 'next': None, 'previous': None, 'results': [{'asset_tag': None, ---&lt;snip&gt;--- 'display': 'cdg-edge-01', ---&lt;snip&gt;--- 'virtual_chassis': None}, {'asset_tag': None, ---&lt;snip&gt;--- 'display': 'cdg-edge-02', ---&lt;snip&gt;--- 'virtual_chassis': None}]} </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">cdg-edge-augment</code> branch has an additional device (<code class="language-plaintext highlighter-rouge">cdg-edge-03</code>) that the <code class="language-plaintext highlighter-rouge">main</code> branch does not.</p> <h3 id="version-control-specific-api-endpoints"><em>Version Control</em>-specific API Endpoints</h3> <p>The <em>Version Control</em> app also has specific API endpoints for querying the following:</p> <ul> <li>Branches</li> <li>Commits</li> <li>Pull Requests</li> <li>Pull Request Comments</li> </ul> <p><img src="../../../static/images/blog_posts/version-control-intro/vc-api-endpoints.png" alt="" /></p> <h2 id="test-drive-the-version-control-app-youself">Test Drive the <em>Version Control</em> App Youself!</h2> <p>This app is still very much in the development phase and is not officially supported (as of the writing of this blog), but you can still test-drive the current functionality in a dev/test environment. The <a href="https://github.com/nautobot/nautobot-plugin-version-control">public repo</a> is available on GitHub. There, you will find <a href="https://github.com/nautobot/nautobot-plugin-version-control#nautobot--dolt">instructions</a> for setting up the dev/test environment.</p> <h2 id="wrapping-up">Wrapping Up</h2> <p>Keeping network automation data clean is important because it affects how well the automation can implement desired network state. Nautobot’s innovative <em>Version Control</em> app provides another layer of data protection to Nautobot’s existing data-protection measures to keep your network automation data clean.</p> <p>Thank you, and have an awesome day!</p> <p>-Tim</p>Tim FiolaOur Nautobots, Rollback! post in late August introduced the community to Dolt databases and the workflow Dolt databases enable in Nautobot. Dolt is enabled in Nautobot via the Version Control app. This post will discuss the value that the Version Control app brings with its Git-like workflows and how those workflows fit into existing Nautobot capabilities to keep network automation data clean.One Parser to Rule All Your Circuit Maintenance Notifications2021-10-05T00:00:00+00:002021-10-05T00:00:00+00:00https://blog.networktocode.com/post/one-parser-to-rule-all-your-circuit-maintenance-notifications<p>A few months ago, Network to Code released two open source projects (more <a href="http://blog.networktocode.com/post/automate-your-circuit-maintenances/">info</a>) to contribute to solving a common problem in modern networks: <strong>understand when a circuit is going through a planned maintenance</strong>. On one side, the <a href="https://github.com/networktocode/circuit-maintenance-parser"><code class="language-plaintext highlighter-rouge">circuit-maintenance-parser</code></a>, to parse notifications into a common data structure, and on the other side, the <a href="https://github.com/nautobot/nautobot-plugin-circuit-maintenance"><code class="language-plaintext highlighter-rouge">nautobot-circuit-maintenance</code> plugin</a>, that uses the parser library to automatically populate the data into <a href="https://nautobot.readthedocs.io/en/stable/">Nautobot</a>. Following months of development, we are happy to announce the release of version <code class="language-plaintext highlighter-rouge">2.0.0</code> of the parser, which comes with a lot of improvements based on existing customer deployments and now <strong>covers 19 different ISPs!</strong></p> <h2 id="circuit-maintenance-parser-library"><code class="language-plaintext highlighter-rouge">circuit-maintenance-parser</code> Library</h2> <p>In the mentioned <a href="http://blog.networktocode.com/post/automate-your-circuit-maintenances/">blog</a>, we acknowledged that we were not the first ones trying to solve this issue. We decided to adopt the format proposed <a href="https://github.com/jda/maintnote-std/blob/master/standard.md">here</a> (using the <code class="language-plaintext highlighter-rouge">iCalendar</code> format) as our gold standard.</p> <p>Now, you could be wondering: why do we need a parser library when there is a well-defined format? The answer is well-known… being just a recommendation it is not fully adopted by all the Network Service Providers (NSPs). So there is still a need to parse arbitrary data formats in order to obtain something conforming to a standard <a href="https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py"><code class="language-plaintext highlighter-rouge">Maintenance</code></a> containing the following attributes:</p> <ul> <li><strong>provider</strong>: identifies the provider of the service that is the subject of the maintenance notification.</li> <li><strong>account</strong>: identifies an account associated with the service that is the subject of the maintenance notification.</li> <li><strong>maintenance_id</strong>: contains text that uniquely identifies the maintenance that is the subject of the notification.</li> <li><strong>circuits</strong>: list of circuits affected by the maintenance notification and their specific impact.</li> <li><strong>status</strong>: defines the overall status or confirmation for the maintenance.</li> <li><strong>start</strong>: timestamp that defines the start date of the maintenance in GMT.</li> <li><strong>end</strong>: timestamp that defines the end date of the maintenance in GMT.</li> <li><strong>stamp</strong>: timestamp that defines the update date of the maintenance in GMT.</li> <li><strong>organizer</strong>: defines the contact information included in the original notification.</li> </ul> <blockquote> <p>Please, refer to the <a href="https://github.com/jda/maintnote-std/blob/master/standard.md">BCOP</a> to more details about these attributes.</p> </blockquote> <p>This library aims to fill the current gap between an ideal representation of a circuit maintenance notification and the current providers’ formats, enabling automation to be built around these notifications.</p> <p>The first version of the <code class="language-plaintext highlighter-rouge">circuit-maintenance-parser</code> had a simple workflow that eventually became a blocker to solve more complex use-cases and this required a new middleware that could combine multiple data using custom logic to process composed notifications, and be able to accommodate future new use-cases. More details about the logic workflow is available in the library <a href="https://github.com/networktocode/circuit-maintenance-parser/blob/develop/README.md#workflow">Readme</a>.</p> <h2 id="supported-providers">Supported Providers</h2> <p>Obviously, one of the key success indicators of the library is how many <code class="language-plaintext highlighter-rouge">Providers</code> are supported, and thanks to multiple examples seen from early adopters, the supported providers list has increased to 19 providers and it’s growing quickly.</p> <ul> <li>AquaComms</li> <li>AWS</li> <li>Cogent</li> <li>Colt</li> <li>EuNetworks</li> <li>GTT</li> <li>HGC</li> <li>Lumen</li> <li>Megaport</li> <li>Momentum</li> <li>NTT</li> <li>PacketFabric</li> <li>Seaborn</li> <li>Sparkle</li> <li>Telia</li> <li>Telstra</li> <li>Turkcell</li> <li>Verizon</li> <li>Zayo</li> </ul> <blockquote> <p>Moreover, the gold format is supported by default with the <code class="language-plaintext highlighter-rouge">GenericProvider</code>, so any NSP that sends the notification with the <code class="language-plaintext highlighter-rouge">iCalendar</code> format is supported by default.</p> </blockquote> <h2 id="how-to-use-it">How to Use It?</h2> <p>The <code class="language-plaintext highlighter-rouge">circuit_maintenance_parser</code> library requires two things:</p> <ul> <li>The <code class="language-plaintext highlighter-rouge">notificationdata</code>: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).</li> <li>The <code class="language-plaintext highlighter-rouge">provider</code> identifier: used to select the required <code class="language-plaintext highlighter-rouge">Provider</code>. Each <code class="language-plaintext highlighter-rouge">Provider</code> contains the logic to process the <code class="language-plaintext highlighter-rouge">notificationdata</code> using associated parsers.</li> </ul> <h3 id="python-library">Python Library</h3> <p>First step is to define the <code class="language-plaintext highlighter-rouge">Provider</code> that we will use to parse the notifications. By default, the <code class="language-plaintext highlighter-rouge">GenericProvider</code> (used when no other provider type is defined) will support parsing of <code class="language-plaintext highlighter-rouge">iCalendar</code> notifications using the recommended format.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">circuit_maintenance_parser</span> <span class="kn">import</span> <span class="n">init_provider</span> <span class="n">generic_provider</span> <span class="o">=</span> <span class="n">init_provider</span><span class="p">()</span> <span class="nb">type</span><span class="p">(</span><span class="n">generic_provider</span><span class="p">)</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">circuit_maintenance_parser</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">GenericProvider</span><span class="s">'&gt; </span></code></pre></div></div> <p>However, usually some <code class="language-plaintext highlighter-rouge">Providers</code> don’t fully implement the standard. Or perhaps some information is missing, for example the <code class="language-plaintext highlighter-rouge">organizer</code> email. We also support custom defined <code class="language-plaintext highlighter-rouge">Providers</code> that can be used to tailor the data extraction based on the notifications your organization receives :</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ntt_provider</span> <span class="o">=</span> <span class="n">init_provider</span><span class="p">(</span><span class="s">"ntt"</span><span class="p">)</span> <span class="nb">type</span><span class="p">(</span><span class="n">ntt_provider</span><span class="p">)</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">circuit_maintenance_parser</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">NTT</span><span class="s">'&gt; </span></code></pre></div></div> <p>Once we have the <code class="language-plaintext highlighter-rouge">Provider</code> ready, we need to initialize the data to process. We call it <code class="language-plaintext highlighter-rouge">NotificationData</code> and can be initialized from a simple content and type or from more complex structures, such as an email.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">circuit_maintenance_parser</span> <span class="kn">import</span> <span class="n">NotificationData</span> <span class="n">raw_data</span> <span class="o">=</span> <span class="sa">b</span><span class="s">"""BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Maint Note//https://github.com/maint-notification// BEGIN:VEVENT SUMMARY:Maint Note Example DTSTART;VALUE=DATE-TIME:20151010T080000Z DTEND;VALUE=DATE-TIME:20151010T100000Z DTSTAMP;VALUE=DATE-TIME:20151010T001000Z UID:42 SEQUENCE:1 X-MAINTNOTE-PROVIDER:example.com X-MAINTNOTE-ACCOUNT:137.035999173 X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415 X-MAINTNOTE-IMPACT:OUTAGE X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-service X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2 X-MAINTNOTE-STATUS:TENTATIVE ORGANIZER;CN="Example NOC":mailto:noone@example.com END:VEVENT END:VCALENDAR """</span> <span class="n">data_to_process</span> <span class="o">=</span> <span class="n">NotificationData</span><span class="p">.</span><span class="n">init_from_raw</span><span class="p">(</span><span class="s">"ical"</span><span class="p">,</span> <span class="n">raw_data</span><span class="p">)</span> <span class="nb">type</span><span class="p">(</span><span class="n">data_to_process</span><span class="p">)</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">circuit_maintenance_parser</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">NotificationData</span><span class="s">'&gt; </span></code></pre></div></div> <p>Finally, we retrieve the maintenances (it is a <code class="language-plaintext highlighter-rouge">List</code> because a notification can contain multiple maintenances) from the data calling the <code class="language-plaintext highlighter-rouge">get_maintenances</code> method from the <code class="language-plaintext highlighter-rouge">Provider</code> instance:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">maintenances</span> <span class="o">=</span> <span class="n">generic_provider</span><span class="p">.</span><span class="n">get_maintenances</span><span class="p">(</span><span class="n">data_to_process</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">maintenances</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">to_json</span><span class="p">())</span> <span class="p">{</span> <span class="s">"account"</span><span class="p">:</span> <span class="s">"137.035999173"</span><span class="p">,</span> <span class="s">"circuits"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"circuit_id"</span><span class="p">:</span> <span class="s">"acme-widgets-as-a-service"</span><span class="p">,</span> <span class="s">"impact"</span><span class="p">:</span> <span class="s">"NO-IMPACT"</span> <span class="p">},</span> <span class="p">{</span> <span class="s">"circuit_id"</span><span class="p">:</span> <span class="s">"acme-widgets-as-a-service-2"</span><span class="p">,</span> <span class="s">"impact"</span><span class="p">:</span> <span class="s">"OUTAGE"</span> <span class="p">}</span> <span class="p">],</span> <span class="s">"end"</span><span class="p">:</span> <span class="mi">1444471200</span><span class="p">,</span> <span class="s">"maintenance_id"</span><span class="p">:</span> <span class="s">"WorkOrder-31415"</span><span class="p">,</span> <span class="s">"organizer"</span><span class="p">:</span> <span class="s">"mailto:noone@example.com"</span><span class="p">,</span> <span class="s">"provider"</span><span class="p">:</span> <span class="s">"example.com"</span><span class="p">,</span> <span class="s">"sequence"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s">"stamp"</span><span class="p">:</span> <span class="mi">1444435800</span><span class="p">,</span> <span class="s">"start"</span><span class="p">:</span> <span class="mi">1444464000</span><span class="p">,</span> <span class="s">"status"</span><span class="p">:</span> <span class="s">"TENTATIVE"</span><span class="p">,</span> <span class="s">"summary"</span><span class="p">:</span> <span class="s">"Maint Note Example"</span><span class="p">,</span> <span class="s">"uid"</span><span class="p">:</span> <span class="s">"42"</span> <span class="p">}</span> </code></pre></div></div> <p>Notice that, either with the <code class="language-plaintext highlighter-rouge">GenericProvider</code> or <code class="language-plaintext highlighter-rouge">NTT</code> provider, we get the same result from the same parsed data, because they are using exactly the same <code class="language-plaintext highlighter-rouge">Processor</code> and <code class="language-plaintext highlighter-rouge">Parser</code>. The only difference is that <code class="language-plaintext highlighter-rouge">NTT Provider</code> will provide some custom default values for NTT in case the notification doesn’t contain this data. In this case, the notification contains all the information, so the custom defaults for <code class="language-plaintext highlighter-rouge">Provider</code> are not used.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ntt_maintenances</span> <span class="o">=</span> <span class="n">ntt_provider</span><span class="p">.</span><span class="n">get_maintenances</span><span class="p">(</span><span class="n">data_to_process</span><span class="p">)</span> <span class="k">assert</span> <span class="n">ntt_maintenances</span> <span class="o">==</span> <span class="n">maintenances</span> </code></pre></div></div> <h3 id="cli">CLI</h3> <p>There is also a <code class="language-plaintext highlighter-rouge">cli</code> entrypoint <code class="language-plaintext highlighter-rouge">circuit-maintenance-parser</code> which offers easy access to the library using few arguments:</p> <ul> <li><code class="language-plaintext highlighter-rouge">data-file</code>: file storing the notification.</li> <li><code class="language-plaintext highlighter-rouge">data-type</code>: <code class="language-plaintext highlighter-rouge">ical</code>, <code class="language-plaintext highlighter-rouge">html</code> or <code class="language-plaintext highlighter-rouge">email</code>, depending on the data type.</li> <li><code class="language-plaintext highlighter-rouge">provider-type</code>: to choose the right <code class="language-plaintext highlighter-rouge">Provider</code>. If empty, the <code class="language-plaintext highlighter-rouge">GenericProvider</code> is used.</li> </ul> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>circuit-maintenance-parser <span class="nt">--data-file</span> <span class="s2">"/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml"</span> <span class="nt">--data-type</span> email <span class="nt">--provider-type</span> zayo Circuit Maintenance Notification <span class="c">#0</span> <span class="o">{</span> <span class="s2">"account"</span>: <span class="s2">"my_account"</span>, <span class="s2">"circuits"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"circuit_id"</span>: <span class="s2">"/OGYX/000000/ /ZYO /"</span>, <span class="s2">"impact"</span>: <span class="s2">"OUTAGE"</span> <span class="o">}</span> <span class="o">]</span>, <span class="s2">"end"</span>: 1601035200, <span class="s2">"maintenance_id"</span>: <span class="s2">"TTN-00000000"</span>, <span class="s2">"organizer"</span>: <span class="s2">"mr@zayo.com"</span>, <span class="s2">"provider"</span>: <span class="s2">"zayo"</span>, <span class="s2">"sequence"</span>: 1, <span class="s2">"stamp"</span>: 1599436800, <span class="s2">"start"</span>: 1601017200, <span class="s2">"status"</span>: <span class="s2">"CONFIRMED"</span>, <span class="s2">"summary"</span>: <span class="s2">"Zayo will implement planned maintenance to troubleshoot and restore degraded span"</span>, <span class="s2">"uid"</span>: <span class="s2">"0"</span> <span class="o">}</span> </code></pre></div></div> <h2 id="how-to-extend-the-library">How to Extend the Library?</h2> <p>Even though the library aims to include support for as many providers as possible, it’s likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new <code class="language-plaintext highlighter-rouge">Provider</code> is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.</p> <p>First step is creating a new file: <code class="language-plaintext highlighter-rouge">circuit_maintenance_parser/parsers/abcde.py</code>. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from <code class="language-plaintext highlighter-rouge">circuit_maintenance_parser.parser</code>. In the example, we only need to import <code class="language-plaintext highlighter-rouge">Html</code> and in the child class implement the methods required by the class, in this case <code class="language-plaintext highlighter-rouge">parse_html()</code> which will return a <code class="language-plaintext highlighter-rouge">dict</code> with all the data that this <code class="language-plaintext highlighter-rouge">Parser</code> can extract. In this case we have to helper methods, <code class="language-plaintext highlighter-rouge">_parse_bs</code> and <code class="language-plaintext highlighter-rouge">_parse_tables</code> that implement the logic to navigate the notification data.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Dict</span> <span class="kn">import</span> <span class="nn">bs4</span> <span class="c1"># type: ignore </span><span class="kn">from</span> <span class="nn">bs4.element</span> <span class="kn">import</span> <span class="n">ResultSet</span> <span class="c1"># type: ignore </span><span class="kn">from</span> <span class="nn">circuit_maintenance_parser.parser</span> <span class="kn">import</span> <span class="n">Html</span> <span class="k">class</span> <span class="nc">HtmlParserABCDE1</span><span class="p">(</span><span class="n">Html</span><span class="p">):</span> <span class="k">def</span> <span class="nf">parse_html</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">soup</span><span class="p">:</span> <span class="n">ResultSet</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{}</span> <span class="bp">self</span><span class="p">.</span><span class="n">_parse_bs</span><span class="p">(</span><span class="n">soup</span><span class="p">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">"b"</span><span class="p">),</span> <span class="n">data</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">_parse_tables</span><span class="p">(</span><span class="n">soup</span><span class="p">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">"table"</span><span class="p">),</span> <span class="n">data</span><span class="p">)</span> <span class="k">return</span> <span class="p">[</span><span class="n">data</span><span class="p">]</span> <span class="k">def</span> <span class="nf">_parse_bs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">btags</span><span class="p">:</span> <span class="n">ResultSet</span><span class="p">,</span> <span class="n">data</span><span class="p">:</span> <span class="n">Dict</span><span class="p">):</span> <span class="p">...</span> <span class="k">def</span> <span class="nf">_parse_tables</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tables</span><span class="p">:</span> <span class="n">ResultSet</span><span class="p">,</span> <span class="n">data</span><span class="p">:</span> <span class="n">Dict</span><span class="p">):</span> <span class="p">...</span> </code></pre></div></div> <p>Next step is to create the new <code class="language-plaintext highlighter-rouge">Provider</code> by defining a new class in <code class="language-plaintext highlighter-rouge">circuit_maintenance_parser/provider.py</code>. This class that inherits from <code class="language-plaintext highlighter-rouge">GenericProvider</code> only needs to define two attributes:</p> <ul> <li><code class="language-plaintext highlighter-rouge">_processors</code>: is a <code class="language-plaintext highlighter-rouge">list</code> of <code class="language-plaintext highlighter-rouge">Processor</code> instances that uses several data <code class="language-plaintext highlighter-rouge">Parsers</code>. In this example, we don’t need to create a new custom <code class="language-plaintext highlighter-rouge">Processor</code> because the combined logic serves well (the most likely case), and we only need to use the new defined <code class="language-plaintext highlighter-rouge">HtmlParserABCDE1</code> and also the generic <code class="language-plaintext highlighter-rouge">EmailDateParser</code> that extract the email date. Also notice that you could have multiple <code class="language-plaintext highlighter-rouge">Processors</code> with different <code class="language-plaintext highlighter-rouge">Parsers</code> in this list, supporting several formats.</li> <li><code class="language-plaintext highlighter-rouge">_default_organizer</code>: this is a default helper to fill the <code class="language-plaintext highlighter-rouge">organizer</code> attribute in the <code class="language-plaintext highlighter-rouge">Maintenance</code> if the information is not part of the original notification.</li> <li><code class="language-plaintext highlighter-rouge">_include_filter</code>: mapping of <code class="language-plaintext highlighter-rouge">data_types</code> to a list of regex expressions that if provided have to match to parse the notification. This feature removes noise from notifications that are received from the same provider, but that are not related to circuit maintenance notifications.</li> <li><code class="language-plaintext highlighter-rouge">_exclude_filter</code>: antagonist mapping to define via regex which are the notifications that must not parsed.</li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ABCDE</span><span class="p">(</span><span class="n">GenericProvider</span><span class="p">):</span> <span class="n">_processors</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">GenericProcessor</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span> <span class="n">CombinedProcessor</span><span class="p">(</span><span class="n">data_parsers</span><span class="o">=</span><span class="p">[</span><span class="n">EmailDateParser</span><span class="p">,</span> <span class="n">HtmlParserABCDE1</span><span class="p">]),</span> <span class="p">]</span> <span class="n">_default_organizer</span> <span class="o">=</span> <span class="s">"noc@abcde.com"</span> <span class="n">_include_filter</span> <span class="o">=</span> <span class="p">{</span><span class="n">EMAIL_HEADER_SUBJECT</span><span class="p">:</span> <span class="p">[</span><span class="s">"Scheduled Maintenance"</span><span class="p">]}</span> </code></pre></div></div> <p>And expose the new <code class="language-plaintext highlighter-rouge">Provider</code> in <code class="language-plaintext highlighter-rouge">circuit_maintenance_parser/__init__.py</code>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">.provider</span> <span class="kn">import</span> <span class="p">(</span> <span class="n">GenericProvider</span><span class="p">,</span> <span class="n">ABCDE</span><span class="p">,</span> <span class="p">...</span> <span class="p">)</span> <span class="n">SUPPORTED_PROVIDERS</span> <span class="o">=</span> <span class="p">(</span> <span class="n">GenericProvider</span><span class="p">,</span> <span class="n">ABCDE</span><span class="p">,</span> <span class="p">...</span> <span class="p">)</span> </code></pre></div></div> <p>Last, but not least, you should update the tests!</p> <ul> <li>Test the new <code class="language-plaintext highlighter-rouge">Parser</code> in <code class="language-plaintext highlighter-rouge">tests/unit/test_parsers.py</code></li> <li>Test the new <code class="language-plaintext highlighter-rouge">Provider</code> logic in <code class="language-plaintext highlighter-rouge">tests/unit/test_e2e.py</code></li> </ul> <p>… adding the necessary data samples in <code class="language-plaintext highlighter-rouge">tests/unit/data/abcde/</code>.</p> <h2 id="whats-next">What’s Next?</h2> <p><strong>Give it a try!</strong>, as the community is growing more and more <code class="language-plaintext highlighter-rouge">Providers</code> are going to be added and you can benefit from all of them. Also, developing a new <code class="language-plaintext highlighter-rouge">Provider</code> or <code class="language-plaintext highlighter-rouge">Parser</code> is straightforward, and you can contribute to the library via <code class="language-plaintext highlighter-rouge">Pull Requests</code> or <code class="language-plaintext highlighter-rouge">Issues</code>, providing notifications samples to develop.</p> <p>As showed in <a href="#how-to-use-it">How to use it?</a>, you can easily integrate it with any automation application only passing the notification data and selecting the <code class="language-plaintext highlighter-rouge">Provider</code> that should be used to parse it, and then do what you want with the structured output.</p> <p>And don’t forget that you could get this integrated with <a href="https://nautobot.readthedocs.io/en/stable/">Nautobot SoT</a> using the <a href="https://github.com/nautobot/nautobot-plugin-circuit-maintenance">Circuit Maintenance Plugin</a>.</p> <p>-Christian</p>Christian AdellA few months ago, Network to Code released two open source projects (more info) to contribute to solving a common problem in modern networks: understand when a circuit is going through a planned maintenance. On one side, the circuit-maintenance-parser, to parse notifications into a common data structure, and on the other side, the nautobot-circuit-maintenance plugin, that uses the parser library to automatically populate the data into Nautobot. Following months of development, we are happy to announce the release of version 2.0.0 of the parser, which comes with a lot of improvements based on existing customer deployments and now covers 19 different ISPs!Agile Planning Horizons (and How to Choose Them) - An Enterprise Guide - Part 22021-10-04T00:00:00+00:002021-10-04T00:00:00+00:00https://blog.networktocode.com/post/agile-planning-horizons-part-2<p>Following from Part 1 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the “Flow” Planning Horizon.</p> <h2 id="overview-flow-planning-horizon">Overview: Flow Planning Horizon</h2> <p>Flow is a “just in time” delivery framework where the work is prioritized and visualized on a Kanban board.</p> <p>The main objective of Flow is <em>THROUGHPUT</em> – the goal is to complete the highest priority work in a continuous flow. This is accomplished by MINIMIZING the work in progress (WIP) so that the team/individual only works on one thing at a time until that work is DONE, then they pull the next highest priority item from the “to-do” stack. Sometimes, work gets blocked waiting for external input or approval – in this case, teams can use a “blocked” column and pull the next highest priority item, but blocked items must be the top priority of any work in flight and be worked as soon as it becomes un-gated.</p> <p>Because work is continuous flow, it does not need to be estimated. Daily standups are not required but often prove to be beneficial – this type of Flow is often referred to as “Scrum-ban”. Retrospectives are not required, but in practice they should be held at some regular interval (e.g., monthly) to review the team’s performance, throughput, and quality, and to address common impediments that should be resolved within the organization.</p> <p>Work is visualized on a Kanban board. The Kanban board uses separate columns to identify work to do next, work in progress, and work completed.</p> <ul> <li>The “To Do” column is a prioritized list of work to be done, where team members pick from the top of the list when taking on new work.</li> <li>The “In Progress” column is a list of the work “in-flight” and being delivered at any given moment. <ul> <li>The “Done” column represents the list of work completed. In Flow, this can also be two columns – an “In Review” column is often used to show work that is completed but requires review and acceptance in order to be considered “Done”. In this case, there will be a “done” or “accepted” state where the work is catalogued for historical record and measurements of the flow rate / velocity.</li> </ul> </li> </ul> <h2 id="the-kanban-board---example">The Kanban board - Example:</h2> <p><img src="../../../static/images/blog_posts/agile-planning-horizons-1/Kanban_board_sample_image.png" alt="Kanban Board Example" /></p> <p>Source: <a href="https://leanicontechnology.com/what-agile-methodology-is-best-for-your-development-teams-in-2020/">[https://leanicontechnology.com/what-agile-methodology-is-best-for-your-development-teams-in-2020/]{.ul}</a></p> <p>Key elements to visualize:</p> <ul> <li>Criteria required for any work item to advance to the next workflow state. For example: Items in “to-do” need to meet some definition of “ready to work” (clear requirements, data inputs identified, etc.) Items in “in progress” need to meet some standard definition of done such as dev complete for all acceptance criteria/requirements, test complete, unit test or code coverage requirements complete &amp; documented, etc.</li> <li>WIP limits – according to the team standard set and agreed upon, work in flight must remain within this limit to ensure smooth and “ASAP” throughput</li> </ul> <h2 id="pros-and-cons">Pros and Cons</h2> <p>Pros – Here are the conditions which lend themselves to a Flow-based planning horizon</p> <ul> <li>The work is routine or repetitive (i.e., the work is well understood, common, frequent)</li> <li>Helpdesk or operational support work - teams dedicated to working defects arising from an established production process or product/application</li> <li>Work that is not plannable, or doesn’t require a detailed/defined pre-planning process (ad hoc work)</li> <li>Work that is simple enough to not require regular and holistic inspect &amp; adapt cycles</li> <li>Teams that are not mature in Agile <ul> <li>Specialized roles are not required (product owner, scrum master, etc.). However, many organizations find it beneficial to have a role that: <ul> <li>Owns prioritization, completeness requirements, and removal of impediments</li> <li>Monitors throughput and delivery results, facilitates continuous improvement</li> </ul> </li> <li>Agile ceremonies are not <em>necessarily</em> required (but often beneficial) such as daily standups, backlog grooming, work planning and commitment, demonstrations of functionality</li> <li>Work doesn’t go through a grooming/refinement and estimation process</li> <li>There is no requirement for strict adherence to team size guidelines per Agile Scrum team best practices (typically 3 to 10 individuals)</li> <li>There is no requirement for truly cross-functional capabilities of the team; often Flow teams can be composed of people who are specialized in one or a few types of work; they don’t require a truly cross-functional skill set in the same way that an Agile Scrum team does.</li> <li>There is no time-boxed cadence (though this can be a double-edged sword and lead to bad habits and inefficient processes unless it is managed and worked with discipline)</li> </ul> </li> </ul> <p>Cons – Here are the conditions which DO NOT lend themselves to a Flow-based planning horizon</p> <ul> <li>Work that is unfamiliar to the organization (“build new” versus “known process”)</li> <li>Work that is complex; work that has upstream or downstream dependencies that must be coordinated and negotiated (i.e., work that needs to be planned for execution both within and outside of the team) <ul> <li>This is especially true for work that is composed of features/epics and strategic initiatives where there are many functional parts that make up a larger whole of value to be delivered</li> </ul> </li> <li>Work that requires multi-discipline skillsets for delivery of work (e.g., UI work, middleware, database, architectural design, security &amp; compliance scope, etc.)</li> <li>Work that requires a high degree of stakeholder, vendor, or other external collaboration and partnership</li> </ul> <h2 id="entry-criteria">Entry Criteria</h2> <p>The entry criteria for teams to adopt a Flow-based Agile Planning Horizon is minimal. You need to know who the individuals working this process are, and the best practice is to have a known and consistent daily capacity for delivering work. The reason for this is so that you can measure velocity, results, and quality, and manage for velocity/quality/productivity improvement over time.</p> <p>With “Scrum-ban”, the team is more well-defined and fits into the accepted range of team size (3 – 10 people). This is because this team will deliver their work via flow/Kanban BUT they will have daily standups to coordinate and plan for the work in flight. In a “Scrum-ban” framework, the team should also participate in Retrospective ceremonies at some defined interval (ie, monthly). The “Scrum-ban” team may also conduct customer/stakeholder demonstrations as needed.</p> <p>Thank you! The next post in this series is a closer look at the Sprinting Agile Planning Horizon.</p> <p>-Matt</p>Matt RemkeFollowing from Part 1 of this series, which provided a high-level overview of three common Agile Planning Horizons, this article provides a closer look at the “Flow” Planning Horizon.Nautobot Relationships Part 22021-09-30T00:00:00+00:002021-09-30T00:00:00+00:00https://blog.networktocode.com/post/nautobot-relationships-part-2<p>In <a href="https://blog.networktocode.com/post/nautobot-relationships-part-1" title="Nautobot Relationships Part 1">Part 1</a> we introduced Nautobot’s new custom defined Relationships feature. Potential use cases were identified, and the process to configure the relationships using the web user interface was outlined. Nautobot Relationships are supported by the REST API (Application Programming Interface) for standard CRUD (Create, Read, Update, Delete) operations and also by the GraphQL API for read-only operations. This post will outline how to leverage the API, for a use case described in Part 1, to programmatically interact with custom-defined relationships.</p> <h2 id="rest">REST</h2> <p>For most systems, REST is the de facto API exposed to clients. It is an HTTP-based service that supports stateless client-server interaction for data retrieval of each model in the Nautobot database. For a more detailed overview, check out the Nautobot docs on <a href="https://nautobot.readthedocs.io/en/latest/rest-api/overview/">REST</a>.</p> <p>HTTP has functions that align to the typical CRUD operations. Nautobot’s RESTful API uses these HTTP functions to interact with the objects in the database. The same operations supported on the GUI are supported on the API. To demonstrate, we’ll create the ‘Circuit IPAddress’ relationship using the API.</p> <h3 id="create-relationship-ipaddress---circuit">Create Relationship: IPAddress - Circuit</h3> <p>The first step when dealing with the API is to consult the endpoint documentation. On the <strong>Home</strong> page, select the <strong>API</strong> button in the bottom right corner.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/home_button_api.png" alt="Nautobot API Documentation" title="Nautobot Swagger API" /></p> <p>The <strong>API</strong> button will redirect to <code class="language-plaintext highlighter-rouge">&lt;server_url&gt;/api/docs/</code> which is the Nautobot API endpoint documentation powered by Swagger. To create a Relationship object a POST operation to the <code class="language-plaintext highlighter-rouge">/extras/relationships/</code> endpoint is required. The docs outline that the <code class="language-plaintext highlighter-rouge">data</code> field for the POST method has a number of required fields, highlighted with a red asterisk.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/rest_post_extras_relationships.png" alt="POST /extras/relationships/" title="REST Create Relationship" /></p> <p>To execute the HTTP POST, we’ll develop a HTTP client using the Python <code class="language-plaintext highlighter-rouge">requests</code> HTTP library, which supports REST and non-REST API communications. Alternatively, the Swagger documentation can be used to test REST API requests directly in the web browser.</p> <p>When creating the Relationship, a valid <code class="language-plaintext highlighter-rouge">slug</code> field consisting of letters, numbers, underscores or hyphens is required to uniquely identify this new object. In the GUI, this field is automatically generated; but in the API it needs to be manually defined. All other fields should be as described in the <a href="https://blog.networktocode.com/post/nautobot-relationships-part-1" title="Nautobot Relationships Part 1">previous post</a>.</p> <blockquote> <p><em>INFO: Nautobot release 1.2 will introduce auto-generated slug fields for the API.</em></p> </blockquote> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server_url</span> <span class="o">=</span> <span class="s">"https://demo.nautobot.com"</span> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">server_url</span><span class="si">}</span><span class="s">/api/extras/relationships/"</span> <span class="n">token</span> <span class="o">=</span> <span class="s">"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"</span> <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Circuit IPAddress"</span><span class="p">,</span> <span class="s">"description"</span><span class="p">:</span> <span class="s">"Custom relationship between circuits and IP addresses"</span><span class="p">,</span> <span class="s">"slug"</span><span class="p">:</span> <span class="s">"circuit-ipaddress"</span><span class="p">,</span> <span class="s">"type"</span><span class="p">:</span> <span class="s">"one-to-many"</span><span class="p">,</span> <span class="s">"source_type"</span><span class="p">:</span> <span class="s">"circuits.circuit"</span><span class="p">,</span> <span class="s">"source_label"</span><span class="p">:</span> <span class="s">"IP Address"</span><span class="p">,</span> <span class="s">"destination_type"</span><span class="p">:</span> <span class="s">"ipam.ipaddress"</span><span class="p">,</span> <span class="s">"destination_label"</span><span class="p">:</span> <span class="s">"Circuit"</span><span class="p">,</span> <span class="p">}</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Authorization"</span><span class="p">:</span> <span class="sa">f</span><span class="s">"Token </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="s">"</span><span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">"POST"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span> </code></pre></div></div> <p>If the object was created successfully, the HTTP return code of 201 is returned, along with the JSON definition of the object.</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2b13240d-cec0-4eb5-a640-ce8f9f47fd9c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://demo.nautobot.com/api/extras/relationships/2b13240d-cec0-4eb5-a640-ce8f9f47fd9c/"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"slug"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit-ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Custom relationship between circuits and IP addresses"</span><span class="p">,</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"one-to-many"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuits.circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"IP Address"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_hidden"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_filter"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipam.ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_hidden"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_filter"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <h3 id="relationship-association">Relationship Association</h3> <p>The Relationship Association is a way to use the new model Relationship. It links instances of the models used in the <code class="language-plaintext highlighter-rouge">source_type</code> and <code class="language-plaintext highlighter-rouge">destination_type</code>. It’s analogous to selecting the remote end of the relationship using a drop-down list on the web GUI.</p> <p>To create the association using the REST API, the <code class="language-plaintext highlighter-rouge">uuid</code> of each instance in the association is configured in the <code class="language-plaintext highlighter-rouge">source_id</code> and <code class="language-plaintext highlighter-rouge">destination_id</code>. The <code class="language-plaintext highlighter-rouge">relationship</code> field can also be configured with a <code class="language-plaintext highlighter-rouge">uuid</code> or, as in the example, a <code class="language-plaintext highlighter-rouge">slug</code> field can be provided within a dictionary, causing Nautobot to do an object lookup.</p> <p>The <code class="language-plaintext highlighter-rouge">uuid</code> values can be retrieved with a GET request to the specific model API endpoint. To simplify the example, we’ll use static object ID’s for each instance in the association. A simple loop will iterate over the two IP address objects to associate each instance with the circuit using the new <code class="language-plaintext highlighter-rouge">circuit-ipaddress</code> Relationship.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server_url</span> <span class="o">=</span> <span class="s">"https://demo.nautobot.com"</span> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">server_url</span><span class="si">}</span><span class="s">/api/extras/relationship-associations/"</span> <span class="n">token</span> <span class="o">=</span> <span class="s">"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"</span> <span class="n">circuit_id</span> <span class="o">=</span> <span class="s">"8b109292-047d-4140-b711-a9524ebb220c"</span> <span class="n">ip_addr1_id</span> <span class="o">=</span> <span class="s">"94a902a3-e063-44b6-889c-d3023b901aff"</span> <span class="n">ip_addr2_id</span> <span class="o">=</span> <span class="s">"8b195702-eca2-4ce3-8d48-68af8d9f5b65"</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Authorization"</span><span class="p">:</span> <span class="sa">f</span><span class="s">"Token </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="s">"</span><span class="p">}</span> <span class="k">for</span> <span class="n">ip_addr_id</span> <span class="ow">in</span> <span class="p">[</span><span class="n">ip_addr1_id</span><span class="p">,</span> <span class="n">ip_addr2_id</span><span class="p">]:</span> <span class="n">payload</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"relationship"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"slug"</span><span class="p">:</span> <span class="s">"circuit-ipaddress"</span><span class="p">,</span> <span class="p">},</span> <span class="s">"source_type"</span><span class="p">:</span> <span class="s">"circuits.circuit"</span><span class="p">,</span> <span class="s">"source_id"</span><span class="p">:</span> <span class="n">circuit_id</span><span class="p">,</span> <span class="s">"destination_type"</span><span class="p">:</span> <span class="s">"ipam.ipaddress"</span><span class="p">,</span> <span class="s">"destination_id"</span><span class="p">:</span> <span class="n">ip_addr_id</span><span class="p">,</span> <span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">"POST"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span> </code></pre></div></div> <p>Once again, if successfully completed, the HTTP return code of 201 is returned, along with the JSON definition of the object(s).</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"830166ef-93af-4eb8-ba56-2fe9e5d15bbe"</span><span class="p">,</span><span class="w"> </span><span class="nl">"relationship"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"00a17433-3f59-437a-a413-3f6e58e7d5f0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"slug"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit-ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"display"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"source_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuits.circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b109292-047d-4140-b711-a9524ebb220c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipam.ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"94a902a3-e063-44b6-889c-d3023b901aff"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"d3d53984-61d5-414a-adaa-eec352baddad"</span><span class="p">,</span><span class="w"> </span><span class="nl">"relationship"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"00a17433-3f59-437a-a413-3f6e58e7d5f0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"slug"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit-ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"display"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"source_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuits.circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b109292-047d-4140-b711-a9524ebb220c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipam.ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b195702-eca2-4ce3-8d48-68af8d9f5b65"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <h3 id="query-relationship-association">Query Relationship Association</h3> <p>In the majority of the cases when using relationships, the client will have an existing object in context, e.g., a circuit. So the query will be from the perspective of that object. The <code class="language-plaintext highlighter-rouge">uuid</code> of the circuit can be used to query along with the Relationship <code class="language-plaintext highlighter-rouge">slug</code> to get back any relationship information. The circuit was the <code class="language-plaintext highlighter-rouge">source_type</code> of the relationship, so the circuit object <code class="language-plaintext highlighter-rouge">uuid</code> is provided as a value into the <code class="language-plaintext highlighter-rouge">source_id</code> query parameter.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server_url</span> <span class="o">=</span> <span class="s">"http://demo.nautobot.com"</span> <span class="n">token</span> <span class="o">=</span> <span class="s">"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"</span> <span class="n">slug</span> <span class="o">=</span> <span class="s">"circuit-ipaddress"</span> <span class="n">circuit_id</span> <span class="o">=</span> <span class="s">"8b109292-047d-4140-b711-a9524ebb220c"</span> <span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">server_url</span><span class="si">}</span><span class="s">/api/extras/relationship-associations/?relationship=</span><span class="si">{</span><span class="n">slug</span><span class="si">}</span><span class="s">&amp;source_id=</span><span class="si">{</span><span class="n">circuit_id</span><span class="si">}</span><span class="s">"</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Authorization"</span><span class="p">:</span> <span class="sa">f</span><span class="s">"Token </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="s">"</span><span class="p">}</span> <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">request</span><span class="p">(</span><span class="s">"GET"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">))</span> </code></pre></div></div> <p>The output return data <code class="language-plaintext highlighter-rouge">count</code> shows two objects, representing the two IP address objects associated with the circuit and their JSON data.</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"count"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="nl">"next"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"> </span><span class="nl">"previous"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w"> </span><span class="nl">"results"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"830166ef-93af-4eb8-ba56-2fe9e5d15bbe"</span><span class="p">,</span><span class="w"> </span><span class="nl">"relationship"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"00a17433-3f59-437a-a413-3f6e58e7d5f0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"slug"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit-ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"display"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"source_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuits.circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b109292-047d-4140-b711-a9524ebb220c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipam.ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"94a902a3-e063-44b6-889c-d3023b901aff"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"d3d53984-61d5-414a-adaa-eec352baddad"</span><span class="p">,</span><span class="w"> </span><span class="nl">"relationship"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"00a17433-3f59-437a-a413-3f6e58e7d5f0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://demo.nautobot.com/api/extras/relationships/00a17433-3f59-437a-a413-3f6e58e7d5f0/"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"slug"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit-ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"display"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Circuit IPAddress"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"source_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"circuits.circuit"</span><span class="p">,</span><span class="w"> </span><span class="nl">"source_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b109292-047d-4140-b711-a9524ebb220c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ipam.ipaddress"</span><span class="p">,</span><span class="w"> </span><span class="nl">"destination_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8b195702-eca2-4ce3-8d48-68af8d9f5b65"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <h2 id="graphql">GraphQL</h2> <p>GraphQL is a recent technology, designed as a query language for APIs and a server-side application for resolving client queries. Refer to the <a href="#resources">Resources</a> section, for some excellent material from my colleague Tim Fiola on GraphQL.</p> <p>In Nautobot, GraphQL supports read-only operations. No updates or modifications can be executed through the GraphQL API. Its power is in complex object data retrieval and the supporting tools for the language, such as GraphiQL, which is an integrated client in Nautobot. GraphiQL can be used to develop queries based on the object model schema to retrieve structured data from the Nautobot GraphQL server. Select the GraphQL button on the bottom-right of the Nautobot <strong>Home</strong> page.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/home_button_graphql.png" alt="Nautobot GraphiQL" title="Nautobot GraphiQL client" /></p> <h3 id="query-relationship-association-1">Query Relationship Association</h3> <p>Nautobot’s GraphQL implementation supports querying custom-defined relationships (and their associations) using the following format, with the hyphens replaced with underscores.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rel_&lt;RELATIONSHIP_SLUG&gt; </code></pre></div></div> <p>Building on the previous example of querying the <code class="language-plaintext highlighter-rouge">circuit-ipaddress</code> Relationship.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rel_circuit_ipaddress </code></pre></div></div> <p>In GraphQL a sample circuit query, filtered with a specific <code class="language-plaintext highlighter-rouge">cid</code> might look as follows.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/graphql_rel_query.png" alt="GraphiQL Query" title="Nautobot GraphiQL Query" /></p> <p>Notice how the <code class="language-plaintext highlighter-rouge">rel_circuit_ipaddress</code> query object has changed the hyphens to underscores. Also note that attributes available to query within the relationship association depend on the type of object.</p> <p>The sample query uses the <code class="language-plaintext highlighter-rouge">interface</code>, <code class="language-plaintext highlighter-rouge">vrf</code> and <code class="language-plaintext highlighter-rouge">address</code> fields to show how the Relationship can be used to extract complex association data for the related IP address object. All GraphQL object type fields can be traversed further to retrieve native or custom-defined relationships. This makes it very easy to get all the required data in just one GraphQL query.</p> <p>An IP address is represented as an <code class="language-plaintext highlighter-rouge">IPAddressType</code> in the GraphQL schema. Schema documentation can be viewed using the <code class="language-plaintext highlighter-rouge">Docs</code> link on the left-hand side of the GraphQL page. The schema for <code class="language-plaintext highlighter-rouge">IPAddressType</code> displays all of the fields available to query.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/IPAddressType_fields_graphql.png" alt="IPAddressType" title="IPAddressType" /></p> <p>Accessing the fields of the model within the relationship using GraphQL is in contrast to the REST API, which returns a reference to the instance. The client then has to make another REST API call to the object endpoint to get the related data.</p> <p>In addition, the GraphQL server response contains only the fields requested in the query. Notice how this is different from a REST response that returns all fields for an endpoint and the client must filter to extract interested fields.</p> <p><img src="../../../static/images/blog_posts/nautobot-relationships-II/graphql_rel_response.png" alt="GraphiQL Response" title="Nautobot GraphiQL response" /></p> <h2 id="conclusion">Conclusion</h2> <p>This series focused on how custom-defined relationships can provide a flexible approach to modeling networks using Nautobot as a source of truth. Use cases facilitated the overall concept; and the various different configuration methods highlight Nautobot’s versatility. Relationships are a key feature in the Nautobot system. If you have any specific use cases please let us know.</p> <p>-Paddy</p> <h2 id="resources">Resources</h2> <ul> <li><a href="https://github.com/nautobot">Nautobot</a></li> <li><a href="https://nautobot.readthedocs.io/en/latest/models/extras/relationship/">Nautobot Relationships</a></li> <li><a href="https://nautobot.readthedocs.io/en/latest/rest-api/overview/">Nautobot REST</a></li> <li><a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">Nautobot GraphQL</a></li> <li><a href="http://blog.networktocode.com/post/nautobot-graphql-requests-via-postman-and-python/">Nautobot GraphQL Requests</a></li> </ul>Paddy KellyIn Part 1 we introduced Nautobot’s new custom defined Relationships feature. Potential use cases were identified, and the process to configure the relationships using the web user interface was outlined. Nautobot Relationships are supported by the REST API (Application Programming Interface) for standard CRUD (Create, Read, Update, Delete) operations and also by the GraphQL API for read-only operations. This post will outline how to leverage the API, for a use case described in Part 1, to programmatically interact with custom-defined relationships.Getting Started with Django2021-09-28T00:00:00+00:002021-09-28T00:00:00+00:00https://blog.networktocode.com/post/getting-started-with-django<p>Django is a popular open-source Python-based framework enabling quick and easy development of web applications. This post will walk through the initial install and setup of Django in order to develop a simple web front end to run a number of network utilities found in Network to Code’s <a href="https://netutils.readthedocs.io/en/latest/">netutils</a> library.</p> <h2 id="django-setup">Django Setup</h2> <p>To get started it is necessary to install the <code class="language-plaintext highlighter-rouge">Django</code> python package using <code class="language-plaintext highlighter-rouge">pip</code>. As a best practice this example installs Django inside a Python virtual environment, which thus should be activated before running <code class="language-plaintext highlighter-rouge">pip install</code>. As shown below, Django has been successfully installed and verified by confirming the installed version of the module.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ntc@ubuntu:~<span class="nv">$ </span><span class="nb">source env</span>/bin/activate <span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects<span class="nv">$ </span>python3 <span class="nt">-m</span> pip <span class="nb">install </span>django </code></pre></div></div> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects<span class="nv">$ </span>pip freeze | <span class="nb">grep </span>Django <span class="nv">Django</span><span class="o">==</span>3.2.6 </code></pre></div></div> <h2 id="create-a-django-project">Create a Django Project</h2> <p>Once Django is installed, the first step to get up and running is to create a Django project. Using the <code class="language-plaintext highlighter-rouge">django-admin startproject</code> command we can create our first Django project, named <code class="language-plaintext highlighter-rouge">network_utilities</code>. In this instance the project has been created in a parent <code class="language-plaintext highlighter-rouge">django-projects</code> directory, which was created manually before starting the Django project and not created by Django itself.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects<span class="nv">$ </span>django-admin startproject network_utilities </code></pre></div></div> <p>Looking inside this directory we can see that Django automatically created a number of subdirectories and Python files. What has been created is a directory store of our main project settings.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects<span class="nv">$ </span>tree <span class="nb">.</span> └── network_utilities ├── manage.py └── network_utilities ├── asgi.py ├── __init__.py ├── settings.py ├── urls.py └── wsgi.py 2 directories, 6 files </code></pre></div></div> <p>Django creates a topmost directory with the same name as our project (i.e., <code class="language-plaintext highlighter-rouge">network_utilities</code>) which contains <code class="language-plaintext highlighter-rouge">manage.py</code>. This is the Python script most used to interact with Django for functions such as creating Django apps, starting the web server, updating database schema, etc.</p> <p>In the project parent directory we have a subdirectory that also has the same name as our project. This directory contains a number of Python scripts used to manage and define project settings:</p> <ul> <li><code class="language-plaintext highlighter-rouge">settings.py</code> - Django project settings including declaring installed Django apps, database connections, etc.</li> <li><code class="language-plaintext highlighter-rouge">urls.py</code> - here we can define the urls for our project</li> <li><code class="language-plaintext highlighter-rouge">asgi.py</code> &amp; <code class="language-plaintext highlighter-rouge">wsgi.py</code> - Web server environment settings</li> </ul> <blockquote> <p>Django’s built-in web server is not suited for production use. When deploying to production you should consider a deployment platform such as wsgi. For the purposes of describing how to get started with Django, the built-in web server will suffice, thus the <code class="language-plaintext highlighter-rouge">asgi.py</code> and <code class="language-plaintext highlighter-rouge">wsgi.py</code> files will not be used.</p> </blockquote> <h2 id="running-django-for-the-first-time">Running Django for the First Time</h2> <p>We can verify the installation by running the Django server using the <code class="language-plaintext highlighter-rouge">manage.py</code> script. For this we should be in the parent <code class="language-plaintext highlighter-rouge">network_utilities</code> directory.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects/network_utilities<span class="nv">$ </span>python manage.py runserver Watching <span class="k">for </span>file changes with StatReloader Performing system checks... System check identified no issues <span class="o">(</span>0 silenced<span class="o">)</span><span class="nb">.</span> You have 18 unapplied migration<span class="o">(</span>s<span class="o">)</span><span class="nb">.</span> Your project may not work properly <span class="k">until </span>you apply the migrations <span class="k">for </span>app<span class="o">(</span>s<span class="o">)</span>: admin, auth, contenttypes, sessions. Run <span class="s1">'python manage.py migrate'</span> to apply them. August 12, 2021 - 16:36:57 Django version 3.2.6, using settings <span class="s1">'network_utilities.settings'</span> Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C. </code></pre></div></div> <p>As can be seen from the output, Django has launched on our localhost on TCP port 8000, i.e., http://127.0.0.1:8000. Browsing to this URL verifies Django has been installed correctly. To stop the server we simply send the CONTROL-C command as instructed above.</p> <p><img src="../../../static/images/blog_posts/Django_Getting_Started/initial_django_page.png" alt="Django Install Verification Page" /></p> <h2 id="a-quick-note-on-django-models">A Quick Note on Django Models</h2> <p>You may have noticed from the output above there is now a <code class="language-plaintext highlighter-rouge">db.sqlite3</code> file included in the project root directory. This was created when we ran the Django <code class="language-plaintext highlighter-rouge">runserver</code> option with the <code class="language-plaintext highlighter-rouge">manage.py</code> script.</p> <p>We were also warned when executing <code class="language-plaintext highlighter-rouge">runserver</code> that there are a number of unapplied migrations. One of the built-in features of Django is an Object-Relational Mapper (ORM) allowing you to define and manage database schemas without the need to understand SQL. Django uses models and migrations to manage your database schema.</p> <p>Models are Python classes used to describe the database schema (i.e. your database table structure), stored in <code class="language-plaintext highlighter-rouge">models.py</code> under each Django application. Migrations are Python scripts that modify your database when changes to the models are made. In this post we are not creating our own models, but it is worth noting that Django by default supports a number of databases, including SQLite (the default, being used here), MySQL, and Oracle, and through third-party integrations can support others (configured in <code class="language-plaintext highlighter-rouge">settings.py</code> <code class="language-plaintext highlighter-rouge">DATABASES</code> dictionary).</p> <p>There are a number of Django apps included by default (listed in <code class="language-plaintext highlighter-rouge">settings.py</code> under <code class="language-plaintext highlighter-rouge">INSTALLED_APPS</code>) which provide “under the hood” Django functionality. The unapplied migrations warning simply states that Django wants to create the necessary database schema for these apps. We can use the <code class="language-plaintext highlighter-rouge">manage.py</code> script to apply these initial migrations:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects/network_utilities<span class="nv">$ </span>python manage.py migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying sessions.0001_initial... OK </code></pre></div></div> <h2 id="django-apps">Django Apps</h2> <p>Django uses the concept of an “application” to provide modularity to our code, one of the built-in advantages of using the framework. A Django app is simply a collection of code that enables a particular function for your web application. When creating the app, a number of Django components are “tied together” within the app, namely:</p> <ul> <li>The view - contains code for the feature we wish to develop</li> <li>The URL pattern - where we define URL’s and map to a view</li> </ul> <p>The view is a Python function with an argument <code class="language-plaintext highlighter-rouge">request</code> which handles the requests for the web page we create. There are other optional components, but to get started, we will just define the view and URL within our Django app for our first simple webpage.</p> <p>Again we use the <code class="language-plaintext highlighter-rouge">manage.py</code> script , this time to create our Django app. Here the app we create is called <code class="language-plaintext highlighter-rouge">network_tools</code>:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects/network_utilities<span class="nv">$ </span>python manage.py startapp network_tools </code></pre></div></div> <p>Django creates a <code class="language-plaintext highlighter-rouge">network_tools</code> directory and a number of files, including <code class="language-plaintext highlighter-rouge">views.py</code> which we will use to create our first view:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects/network_utilities<span class="nv">$ </span>tree <span class="nb">.</span> ├── db.sqlite3 ├── manage.py ├── network_tools │ ├── admin.py │ ├── apps.py │ ├── __init__.py │ ├── migrations │ │ └── __init__.py │ ├── models.py │ ├── tests.py │ └── views.py └── network_utilities ├── asgi.py ├── __init__.py ├── __pycache__ │ ├── __init__.cpython-38.pyc │ ├── settings.cpython-38.pyc │ ├── urls.cpython-38.pyc │ └── wsgi.cpython-38.pyc ├── settings.py ├── urls.py └── wsgi.py 4 directories, 18 files </code></pre></div></div> <h2 id="views--urls">Views &amp; URLs</h2> <p>In order to create our first view it is necessary to declare our Django app in the <code class="language-plaintext highlighter-rouge">INSTALLED_APPS</code> list in <code class="language-plaintext highlighter-rouge">settings.py</code> (in the <code class="language-plaintext highlighter-rouge">network_utilities</code> sub-directory).</p> <p><code class="language-plaintext highlighter-rouge">network_utilities/settings.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Application definition </span> <span class="n">INSTALLED_APPS</span> <span class="o">=</span> <span class="p">[</span> <span class="s">'django.contrib.admin'</span><span class="p">,</span> <span class="s">'django.contrib.auth'</span><span class="p">,</span> <span class="s">'django.contrib.contenttypes'</span><span class="p">,</span> <span class="s">'django.contrib.sessions'</span><span class="p">,</span> <span class="s">'django.contrib.messages'</span><span class="p">,</span> <span class="s">'django.contrib.staticfiles'</span><span class="p">,</span> <span class="s">'network_tools'</span><span class="p">,</span> <span class="p">]</span> </code></pre></div></div> <p>A simple home page can be created in the <code class="language-plaintext highlighter-rouge">views.py</code> file under the <code class="language-plaintext highlighter-rouge">network_tools</code> app. Here we create a simple function called <code class="language-plaintext highlighter-rouge">home</code> with a single argument, <code class="language-plaintext highlighter-rouge">request</code>. This argument will capture the HTTP requests which we will use later. The function returns a welcome message using Django’s <code class="language-plaintext highlighter-rouge">HttpResponse</code>, which we import on line 1.</p> <p><code class="language-plaintext highlighter-rouge">network_tools/views.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span> <span class="k">def</span> <span class="nf">home</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="s">"Welcome to NTC netutils Django demo"</span><span class="p">)</span> </code></pre></div></div> <p>Next we must map a URL to this view. Currently we have one <code class="language-plaintext highlighter-rouge">urls.py</code> file in the project settings directory i.e. <code class="language-plaintext highlighter-rouge">network_utilities/network_utilities</code>. It is possible to declare all our URL’s here but we also have an option to create a per-Django-application <code class="language-plaintext highlighter-rouge">urls.py</code>. As well as modularity this approach allows us to have a common prefix for each app. This is the approach we will take allowing us to create a <code class="language-plaintext highlighter-rouge">/netutils</code> prefix as we add webpages. We will create this prefix later when adding pages for the <code class="language-plaintext highlighter-rouge">netutils</code> utilities, but first we will setup the project and application <code class="language-plaintext highlighter-rouge">urls.py</code> for the home page.</p> <p>In the project <code class="language-plaintext highlighter-rouge">urls.py</code> we map the home page URL (signified via a blank path <code class="language-plaintext highlighter-rouge">''</code> ) to our Django app by including <code class="language-plaintext highlighter-rouge">network_tools.urls</code>.</p> <p><code class="language-plaintext highlighter-rouge">network_utilities/urls.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="kn">import</span> <span class="n">admin</span> <span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span> <span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span> <span class="n">path</span><span class="p">(</span><span class="s">'admin/'</span><span class="p">,</span> <span class="n">admin</span><span class="p">.</span><span class="n">site</span><span class="p">.</span><span class="n">urls</span><span class="p">),</span> <span class="n">path</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="n">include</span><span class="p">(</span><span class="s">'network_tools.urls'</span><span class="p">))</span> <span class="p">]</span> </code></pre></div></div> <p>We then create a <code class="language-plaintext highlighter-rouge">urls.py</code> file in the application directory (<code class="language-plaintext highlighter-rouge">network_utilities/network_tools</code>) in order to map the URL to our view, you can see that the path (<code class="language-plaintext highlighter-rouge">''</code>) is the same in both files.</p> <p><code class="language-plaintext highlighter-rouge">network_tools/urls.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span> <span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">views</span> <span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span> <span class="n">path</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">home</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'home'</span><span class="p">),</span> <span class="p">]</span> </code></pre></div></div> <p>Now if we run the Django server again and browse to http://127.0.0.1:8000 we will be presented with the welcome message and we have created our first Django webpage.</p> <p><img src="../../../static/images/blog_posts/Django_Getting_Started/welcome.png" alt="Welcome Page" /></p> <h2 id="building-the-web-app">Building the Web App</h2> <p>Having covered the basics of Django we can build our web frontend for the NTC netutils library (<code class="language-plaintext highlighter-rouge">python -m pip install netutils</code>)</p> <p>So far we have created a simple webpage with a welcome message, now we can use Django templates to write HTML code to develop the page further. The templates support an inheritance model whereby we can ensure that each page we create has a similar look and feel. To start we will create a <code class="language-plaintext highlighter-rouge">base.html</code> file, from which all other pages will inherit.</p> <p>First we must create a <code class="language-plaintext highlighter-rouge">templates</code> directory inside our app directory. Django by default will look in this directory for the <code class="language-plaintext highlighter-rouge">base.html</code> file. This directory should also have a sub-directory with the same name as the app (<code class="language-plaintext highlighter-rouge">network_tools</code>) where HTML files for all the app pages will be stored. Similarly a <code class="language-plaintext highlighter-rouge">static</code> directory with a <code class="language-plaintext highlighter-rouge">network_tools</code> sub directory is created to store an image that will be displayed on each page inherited from <code class="language-plaintext highlighter-rouge">base.html</code>.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> network_utilities/network_tools<span class="nv">$ </span>tree ├── static │ └── network_tools │ └── ntc-logo.png ├── templates │ ├── base.html │ └── network_tools │ └── home.html </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">base.html</code> file includes 2 blocks (<code class="language-plaintext highlighter-rouge">title</code> and <code class="language-plaintext highlighter-rouge">body</code>) which act as placeholders for subsequent pages to display page specific content. The header also includes a PNG image (the Network to Code logo); each page that extends from <code class="language-plaintext highlighter-rouge">base.html</code> will display this image. In order to load the image the <code class="language-plaintext highlighter-rouge">load static</code> instruction is necessary i.e. line 1.</p> <p><code class="language-plaintext highlighter-rouge">templates/base.html</code></p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% load static %} <span class="cp">&lt;!DOCTYPE html&gt;</span> <span class="nt">&lt;html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">&gt;</span> <span class="nt">&lt;head&gt;</span> <span class="nt">&lt;meta</span> <span class="na">charset=</span><span class="s">"UTF-8"</span><span class="nt">&gt;</span> <span class="nt">&lt;title&gt;</span>{% block title %}{% endblock %}<span class="nt">&lt;/title&gt;</span> <span class="nt">&lt;img</span> <span class="na">src=</span><span class="s">"{% static 'network_tools/ntc-logo.png' %}"</span> <span class="na">width=</span><span class="s">"200px"</span><span class="nt">&gt;</span> <span class="nt">&lt;/head&gt;</span> <span class="nt">&lt;body&gt;</span> {% block body %} {% endblock %} <span class="nt">&lt;/body&gt;</span> <span class="nt">&lt;/html&gt;</span> </code></pre></div></div> <p>We can now update the home page by creating a <code class="language-plaintext highlighter-rouge">home.html</code> in the <code class="language-plaintext highlighter-rouge">templates/network_tools</code> directory. This HTML file extends from <code class="language-plaintext highlighter-rouge">base.html</code> and provides inputs to the <code class="language-plaintext highlighter-rouge">title</code> and <code class="language-plaintext highlighter-rouge">body</code> blocks.</p> <p><code class="language-plaintext highlighter-rouge">templates/network_tools/home.html</code></p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% extends "base.html" %} {% block title %}Welcome{% endblock %} {% block body %} <span class="nt">&lt;h1&gt;</span> Welcome to NTC netutils Django front end demo<span class="nt">&lt;/h1&gt;</span> <span class="nt">&lt;body&gt;</span> <span class="nt">&lt;p&gt;</span> This is the webpage for running NTC netutils utilities. <span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;/body&gt;</span> {% endblock %} </code></pre></div></div> <p>In order to utilise these HTML templates we must update our view. The Django <code class="language-plaintext highlighter-rouge">render</code> function is now used instead of directly constructing an <code class="language-plaintext highlighter-rouge">HttpResponse</code>, and thus we can remove the <code class="language-plaintext highlighter-rouge">from django.http import HttpResponse</code> statement.</p> <p><code class="language-plaintext highlighter-rouge">network_tools/views.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span> <span class="k">def</span> <span class="nf">home</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="s">"network_tools/home.html"</span><span class="p">)</span> </code></pre></div></div> <p>We now have a home page which inherits the NTC logo from base.html.</p> <p><img src="../../../static/images/blog_posts/Django_Getting_Started/welcome_ntc_logo.png" alt="Welcome with NTC logo" /></p> <h2 id="integrating-netutils">Integrating netutils</h2> <p>NTC’s <code class="language-plaintext highlighter-rouge">netutils</code> Python library is a collection of tools to help with your network automation projects, installed using ‘pip’:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">env</span><span class="o">)</span> ntc@ubuntu:~/django-projects<span class="nv">$ </span>python3 <span class="nt">-m</span> pip <span class="nb">install </span>netutils </code></pre></div></div> <p>For the purposes of this post we will create a frontend for 2 of the ‘netutils’ tools:</p> <ul> <li>TCP Ping - verify if a host is listening on a particular TCP port</li> <li>IP Address Utilities - e.g., netmask to CIDR notation and vice-versa</li> </ul> <p>As with our home page, it is necessary to update a number of files in order to create each new page as follows:</p> <ul> <li><code class="language-plaintext highlighter-rouge">ping.html</code> - a new file in the <code class="language-plaintext highlighter-rouge">templates/network_tools</code> directory</li> <li><code class="language-plaintext highlighter-rouge">views.py</code> - define our view for each page</li> <li><code class="language-plaintext highlighter-rouge">urls.py</code> - In both the project and application directories allows us to map a URL to our new view</li> </ul> <p>First we will create a link on our home page to our new page for the <code class="language-plaintext highlighter-rouge">netutils</code> tool. This is configured in <code class="language-plaintext highlighter-rouge">home.html</code> we created previously, we have added a sub header and HREF for the “Ping” page to specify a link destination.</p> <p><code class="language-plaintext highlighter-rouge">templates/network_tools/home.html</code></p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% extends "base.html" %} {% block title %}Welcome{% endblock %} {% block body %} <span class="nt">&lt;h1&gt;</span> Welcome to NTC netutils Django front end demo.<span class="nt">&lt;/h1&gt;</span> <span class="nt">&lt;body&gt;</span> <span class="nt">&lt;p&gt;</span> This is the webpage for running NTC netutils utilities. <span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;h2&gt;</span> Supported Netutils Functions:<span class="nt">&lt;/h2&gt;</span> <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"{%url 'ping' %}"</span><span class="nt">&gt;</span>Ping<span class="nt">&lt;/a&gt;</span> <span class="nt">&lt;/body&gt;</span> {% endblock %} </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">ping.html</code> template will inherit from <code class="language-plaintext highlighter-rouge">base.html</code> as before. We can now complete the page specific details using the 2 blocks declared in <code class="language-plaintext highlighter-rouge">base.html</code>, <code class="language-plaintext highlighter-rouge">title</code> and <code class="language-plaintext highlighter-rouge">body</code>. The <code class="language-plaintext highlighter-rouge">title</code> simply labels our browser tab, with ‘TCP Ping’ in this instance. The <code class="language-plaintext highlighter-rouge">body</code> block allows us to code the input fields necessary for the netutils TCP Ping utility, note in the body we specify the HTTP method verb which we will use in the view. The netutils ping tool accepts 2 arguments (IP Address and TCP Port number) requiring a form field for each argument, both of which are given a name (“ip” and “number”) which can be referenced in a Django view. We also create a submit button to allow the user to execute the TCP Ping utility.</p> <p>The HTML code includes input validation on the TCP port to ensure only an integer in the valid TCP port range is accepted. Also a <code class="language-plaintext highlighter-rouge">test_result</code> variable is displayed which will be declared in our Django view. A link back to the Home page is added at the bottom of the page.</p> <p><code class="language-plaintext highlighter-rouge">templates/network_tools/ping.html</code></p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% extends "base.html" %} {% block title %}TCP Ping{% endblock %} {% block body %} <span class="nt">&lt;h2&gt;</span> TCP Ping <span class="nt">&lt;/h2&gt;</span> <span class="nt">&lt;p&gt;</span> Verifies whether a TCP port is open on a given IP address <span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;form</span> <span class="na">action=</span><span class="s">"/ping/"</span> <span class="na">method=</span><span class="s">"get"</span><span class="nt">&gt;</span> <span class="nt">&lt;label</span> <span class="na">for=</span><span class="s">""</span><span class="nt">&gt;</span>Please enter Host IP Address:<span class="nt">&lt;/label&gt;</span> <span class="nt">&lt;input</span> <span class="na">name=</span><span class="s">"ip"</span><span class="nt">&gt;</span> <span class="nt">&lt;div&gt;&lt;label</span> <span class="na">for=</span><span class="s">""</span><span class="nt">&gt;</span>Please enter TCP Port:<span class="nt">&lt;/label&gt;</span> <span class="nt">&lt;input</span> <span class="na">type=</span><span class="s">"number"</span> <span class="na">name=</span><span class="s">"port"</span> <span class="na">min=</span><span class="s">"0"</span> <span class="na">max=</span><span class="s">"65535"</span><span class="nt">&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;div&gt;&lt;button</span> <span class="na">type=</span><span class="s">"submit"</span><span class="nt">&gt;</span>Submit<span class="nt">&lt;/button&gt;&lt;/div&gt;</span> <span class="nt">&lt;/form&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"result"</span><span class="nt">&gt;</span>{{test_result|linebreaks}}<span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"{%url 'home' %}"</span><span class="nt">&gt;</span>home<span class="nt">&lt;/a&gt;</span> {% endblock %} </code></pre></div></div> <p>In <code class="language-plaintext highlighter-rouge">views.py</code> we declare a new function to define the code necessary to execute the ping tool. The required functions from the netutils library are imported, as shown on line 2 and 3. The request object contains the data entered by the user, i.e. the ‘ip’ and ‘port’ parameters specified in <code class="language-plaintext highlighter-rouge">ping.html</code>.</p> <p>The “ping” function includes input validation to verify the user input. First it verifies an IP Address has been entered, in this instance we use the <code class="language-plaintext highlighter-rouge">get</code> method of the <code class="language-plaintext highlighter-rouge">request.GET</code> object to query the <code class="language-plaintext highlighter-rouge">'ip'</code> key. If an IP Address has been entered the netutils <code class="language-plaintext highlighter-rouge">is_ip</code> method is used to ensure the IP Address is in a valid format. This input validation is in addition to verification of the TCP port number configured in the HTML template which is enforced by the browser rather than the <code class="language-plaintext highlighter-rouge">views.py</code> function.</p> <p>Having validated the user input, the netutils <code class="language-plaintext highlighter-rouge">tcp_ping</code> function is called and the result stored in a <code class="language-plaintext highlighter-rouge">test_result</code> dictionary. The function <code class="language-plaintext highlighter-rouge">return</code> statement calls Django <code class="language-plaintext highlighter-rouge">render</code> to render the <code class="language-plaintext highlighter-rouge">ping.html</code> template and passes in the <code class="language-plaintext highlighter-rouge">test_result</code> dictionary.</p> <p><code class="language-plaintext highlighter-rouge">network_tools/views.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span> <span class="kn">from</span> <span class="nn">netutils.ping</span> <span class="kn">import</span> <span class="n">tcp_ping</span> <span class="kn">from</span> <span class="nn">netutils.ip</span> <span class="kn">import</span> <span class="n">is_ip</span><span class="p">,</span> <span class="n">cidr_to_netmask</span><span class="p">,</span> <span class="n">netmask_to_cidr</span><span class="p">,</span> <span class="n">is_netmask</span> <span class="k">def</span> <span class="nf">home</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="s">"network_tools/home.html"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">ping</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="n">test_result</span> <span class="o">=</span> <span class="p">{}</span> <span class="k">if</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'ip'</span><span class="p">)</span> <span class="o">==</span> <span class="s">''</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'test_result'</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'Please enter an IP Address </span><span class="si">{</span><span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="si">}</span><span class="s">'</span> <span class="k">elif</span> <span class="s">'ip'</span> <span class="ow">in</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">:</span> <span class="k">if</span> <span class="n">is_ip</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">[</span><span class="s">'ip'</span><span class="p">])</span> <span class="ow">is</span> <span class="bp">False</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'test_result'</span><span class="p">]</span> <span class="o">=</span> <span class="s">'Please enter a valid IP Address'</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">"network_tools/ping.html"</span><span class="p">,</span> <span class="n">test_result</span><span class="p">)</span> <span class="n">host_ip</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">[</span><span class="s">'ip'</span><span class="p">]</span> <span class="n">host_port</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">[</span><span class="s">'port'</span><span class="p">]</span> <span class="n">ping_result</span> <span class="o">=</span> <span class="n">tcp_ping</span><span class="p">(</span><span class="n">host_ip</span><span class="p">,</span> <span class="n">host_port</span><span class="p">)</span> <span class="k">if</span> <span class="n">ping_result</span> <span class="ow">is</span> <span class="bp">True</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'test_result'</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"Success: Port </span><span class="si">{</span><span class="n">host_port</span><span class="si">}</span><span class="s"> is Open on host </span><span class="si">{</span><span class="n">host_ip</span><span class="si">}</span><span class="s">"</span> <span class="k">else</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'test_result'</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"Failure: Cannot open connection to Port </span><span class="si">{</span><span class="n">host_port</span><span class="si">}</span><span class="s"> on host </span><span class="si">{</span><span class="n">host_ip</span><span class="si">}</span><span class="s">"</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">"network_tools/ping.html"</span><span class="p">,</span> <span class="n">test_result</span><span class="p">)</span> </code></pre></div></div> <p>Finally we must update the project and application <code class="language-plaintext highlighter-rouge">urls.py</code> to map a URL to our django view.</p> <p>In the project <code class="language-plaintext highlighter-rouge">urls.py</code> we can now declare a <code class="language-plaintext highlighter-rouge">netutils</code> prefix which will be prepended to pages for this Django app.</p> <p><code class="language-plaintext highlighter-rouge">network_utilities/urls.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="kn">import</span> <span class="n">admin</span> <span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span><span class="p">,</span> <span class="n">include</span> <span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span> <span class="n">path</span><span class="p">(</span><span class="s">'admin/'</span><span class="p">,</span> <span class="n">admin</span><span class="p">.</span><span class="n">site</span><span class="p">.</span><span class="n">urls</span><span class="p">),</span> <span class="n">path</span><span class="p">(</span><span class="s">'netutils/'</span><span class="p">,</span> <span class="n">include</span><span class="p">(</span><span class="s">'network_tools.urls'</span><span class="p">))</span> <span class="p">]</span> </code></pre></div></div> <p>In the application <code class="language-plaintext highlighter-rouge">urls.py</code> we map the URL to the view.</p> <p><code class="language-plaintext highlighter-rouge">network_tools/urls.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span> <span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">views</span> <span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span> <span class="n">path</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">home</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"home"</span><span class="p">),</span> <span class="n">path</span><span class="p">(</span><span class="s">'ping/'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">ping</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"ping"</span><span class="p">),</span> <span class="p">]</span> </code></pre></div></div> <p>We now have a new webpage allowing the user to validate if an IP Address is listening on a particular TCP port. The input validation configured will ensure a message is displayed if an invalid IP Address or TCP Port number is entered. In the example output below we entered valid details and were returned a ‘Success’ message.</p> <p><img src="../../../static/images/blog_posts/Django_Getting_Started/TCP_Ping.png" alt="TCP Ping Page" /></p> <h2 id="adding-additional-pages">Adding Additional Pages</h2> <p>This completes all the necessary steps to enable a Django frontend on one of Network to Code’s <code class="language-plaintext highlighter-rouge">netutils</code> functions. In order to create additional pages (e.g. the IP Address utilities) we follow the same process:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Add a link to the new page on the home page in home.html. 2. Add an HTML template for our new page, e.g., ip_addresses.html. 3. Define the view in views.py, e.g., an ip_addr(request) function plus necessary code logic. 4. Update the application-specific urls.py to map a url to the new view. </code></pre></div></div> <p>At this stage most of the work is in <code class="language-plaintext highlighter-rouge">views.py</code> and the application specific <code class="language-plaintext highlighter-rouge">urls.py</code>. Below is shown the changes to these files to add the CIDR/Netmask conversion tool.</p> <p><code class="language-plaintext highlighter-rouge">templates/network_tools/ip_address.html</code></p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% block body %} <span class="nt">&lt;h2&gt;</span> cidr_to_netmask<span class="nt">&lt;/h2&gt;</span> <span class="nt">&lt;p&gt;</span> Creates a decimal format of a CIDR value. <span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;form</span> <span class="na">action=</span><span class="s">"/ip_addr"</span> <span class="na">method=</span><span class="s">"get"</span><span class="nt">&gt;</span> <span class="nt">&lt;div&gt;&lt;label</span> <span class="na">for=</span><span class="s">""</span><span class="nt">&gt;</span>Please enter CIDR value:<span class="nt">&lt;/label&gt;</span> <span class="nt">&lt;input</span> <span class="na">type=</span><span class="s">"number"</span> <span class="na">name=</span><span class="s">"cidr"</span> <span class="na">min=</span><span class="s">"1"</span> <span class="na">max=</span><span class="s">"32"</span><span class="nt">&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;div&gt;&lt;button</span> <span class="na">type=</span><span class="s">"submit"</span><span class="nt">&gt;</span>Submit<span class="nt">&lt;/button&gt;&lt;/div&gt;</span> <span class="nt">&lt;/form&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"result"</span><span class="nt">&gt;</span>{{netmask_result|linebreaks}}<span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;h2&gt;</span> netmask_to_cidr<span class="nt">&lt;/h2&gt;</span> <span class="nt">&lt;p&gt;</span> Creates a CIDR notation of a given subnet mask in decimal format. <span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;form</span> <span class="na">action=</span><span class="s">"/ip_addr"</span> <span class="na">method=</span><span class="s">"get"</span><span class="nt">&gt;</span> <span class="nt">&lt;div&gt;&lt;label</span> <span class="na">for=</span><span class="s">""</span><span class="nt">&gt;</span>Please enter a netmask value:<span class="nt">&lt;/label&gt;</span> <span class="nt">&lt;input</span> <span class="na">name=</span><span class="s">"netmask"</span><span class="nt">&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;div&gt;&lt;button</span> <span class="na">type=</span><span class="s">"submit"</span><span class="nt">&gt;</span>Submit<span class="nt">&lt;/button&gt;&lt;/div&gt;</span> <span class="nt">&lt;/form&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"result"</span><span class="nt">&gt;</span>{{cidr_result|linebreaks}}<span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"{%url 'home' %}"</span><span class="nt">&gt;</span>home<span class="nt">&lt;/a&gt;</span> {% endblock %} </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">network_tools/views.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">ip_addr</span><span class="p">(</span><span class="n">request</span><span class="p">):</span> <span class="n">test_result</span> <span class="o">=</span> <span class="p">{}</span> <span class="k">if</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'cidr'</span><span class="p">)</span> <span class="o">==</span> <span class="s">''</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'test_result'</span><span class="p">]</span> <span class="o">=</span> <span class="s">'Please a valid CIDR value [1-24]'</span> <span class="k">elif</span> <span class="s">'cidr'</span> <span class="ow">in</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">:</span> <span class="n">cidr</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">[</span><span class="s">'cidr'</span><span class="p">])</span> <span class="n">netmask_result</span> <span class="o">=</span> <span class="n">cidr_to_netmask</span><span class="p">(</span><span class="n">cidr</span><span class="p">)</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'netmask_result'</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"Netmask for CIDR value </span><span class="si">{</span><span class="n">cidr</span><span class="si">}</span><span class="s"> is </span><span class="si">{</span><span class="n">netmask_result</span><span class="si">}</span><span class="s">"</span> <span class="k">elif</span> <span class="s">'netmask'</span> <span class="ow">in</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">:</span> <span class="n">netmask</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="n">GET</span><span class="p">[</span><span class="s">'netmask'</span><span class="p">]</span> <span class="k">if</span> <span class="n">is_netmask</span><span class="p">(</span><span class="n">netmask</span><span class="p">)</span> <span class="ow">is</span> <span class="bp">False</span><span class="p">:</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'cidr_result'</span><span class="p">]</span> <span class="o">=</span> <span class="s">'Please enter a valid Netmask'</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">"network_tools/ip_address.html"</span><span class="p">,</span> <span class="n">test_result</span><span class="p">)</span> <span class="n">cidr_value</span> <span class="o">=</span> <span class="n">netmask_to_cidr</span><span class="p">(</span><span class="n">netmask</span><span class="p">)</span> <span class="n">test_result</span><span class="p">[</span><span class="s">'cidr_result'</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"CIDR Value for netmask </span><span class="si">{</span><span class="n">netmask</span><span class="si">}</span><span class="s"> is </span><span class="si">{</span><span class="n">cidr_value</span><span class="si">}</span><span class="s">"</span> <span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">"network_tools/ip_address.html"</span><span class="p">,</span> <span class="n">test_result</span><span class="p">)</span> </code></pre></div></div> <p>This gives us 2 additional tools to convert CIDR value to Netmask and vice-versa.</p> <p><img src="../../../static/images/blog_posts/Django_Getting_Started/IP_Address.png" alt="IP Address Page" /></p> <h2 id="summary">Summary</h2> <p>We have just scratched the surface of Django, but hopefully this will give you an understanding of how to quickly get started. I hope you find this blog post useful.</p> <p>-Nicholas Davey</p>Nicholas DaveyDjango is a popular open-source Python-based framework enabling quick and easy development of web applications. This post will walk through the initial install and setup of Django in order to develop a simple web front end to run a number of network utilities found in Network to Code’s netutils library.