Jekyll2022-01-18T13:24:33+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comIntro to Pandas (Part 2) - Exploratory data analysis for network traffic2022-01-18T00:00:00+00:002022-01-18T00:00:00+00:00https://blog.networktocode.com/post/exploratory-data-analysis-for-network-traffic<p>Data analytics is an important skill for every engineer and even more the Network Engineer that goes through large amounts of data for troubleshooting. Networking companies have been moving towards data science integrations for appliances and software. The <a href="https://hostingjournalist.com/arista-introduces-eos-network-data-lake-for-data-driven-cloud-networking/">Arista EOS Network Data Lake</a> is a characteristic example where Artificial Intelligence and Machine Learning are used to analyze data from different resources and lead to actionable decisions.</p> <p>This blog aims to develop these skills, and it is a part of a series related to data analysis for Network Engineers. The <a href="https://blog.networktocode.com/post/introduction-to-pandas-for-network-development/">first part</a> was a detailed introduction on how to use Pandas, a powerful Python Data Science framework, to analyze networking data. The <a href="https://blog.networktocode.com/post/jupyter-notebooks-for-development/">second part</a> included instructions on how to run the code in these blogs using Jupyter notebooks and the Poetry virtual environment. This third blog is going deeper into how we can explore black-box networking data with a powerful analysis technique, Exploratory Data Analysis (EDA). Naturally, we will be using Pandas and Jupyter notebooks in our examples.</p> <h2 id="exploratory-data-analysis-eda">Exploratory Data Analysis (EDA)</h2> <p><a href="https://www.itl.nist.gov/div898/handbook/eda/eda.htm">Exploratory Data Analysis</a> (EDA) is an approach/philosophy for data analysis that employs a variety of statistical and graphical techniques to make sense of any type of black-box data.</p> <h3 id="goals-of-eda">Goals of EDA</h3> <p>EDA aims at accomplishing the following goals:</p> <ul> <li><em>Tailor a good fitting</em>: Matching your data as closely as possible to a distribution described by a mathematical expression has several benefits, such as predicting the next failure in your network.</li> <li><em>Find outliers</em>: Outliers are these odd data points that lie outside of a group of data. An example is a web server farm where all port requests are aimed at specific ports and every once in a while a random port is requested. An outlier may be caused by error or intentional testing and even adversarial attack behavior.</li> <li><em>Create a list of ranked important factors</em>: Removing unwanted features or finding the most important ones is called <code class="language-plaintext highlighter-rouge">dimensionality reduction</code> in data science terms. For example, with EDA you will be able to distinguish using statistical metrics a subset of the most important features or appliances that may affect your network performance, outages, and errors.</li> <li><em>Discover optimal settings</em>: How many times have you wondered how it would be if you could fine-tune your BGP timers, MTUs, or bandwidth allocation and not just guess these values? EDA helps discover the best value for networking settings.</li> </ul> <h3 id="why-eda">Why EDA</h3> <p>EDA has been proven an invaluable tool for Data Scientists, why not for Network Engineers? We gather a lot of network data, such as packet captures, syslogs, etc., that we do not know how to make sense of or how to mine their value. Even though we use out-of-box data analytics tools, such as Splunk, the insight of the Network Engineer that stems from building a model and processing raw data, is invaluable.</p> <h3 id="how-to-implement-eda">How to Implement EDA</h3> <p>To implement EDA, you need tools that you probably use in your day-to-day network operations and did not know they were part of EDA:</p> <ul> <li><em>Strong graphical analysis</em>: from single variable plots to time series, and multi-variable plots, there is a graphical tool in EDA that fits your problem.</li> <li><em>Statistics</em>: this may include hypothesis testing, calculations of summary statistics, metrics for scale, and the shape of your data.</li> </ul> <p>We will explore these techniques with a network dataset in the next section.</p> <h2 id="eda-for-network-data">EDA for Network Data</h2> <p>In this section, we will review data preprocessing with graphical and statistical analysis EDA techniques.</p> <h3 id="dataset">Dataset</h3> <p>The dataset we will use is a 5GB packet capture of Operating System (OS) scans from the <a href="https://www.kaggle.com/ymirsky/network-attack-dataset-kitsune">Kitsune Network Attack Dataset</a>. You will find the code referenced below in the <a href="https://github.com/mundruid/pandas-blog/blob/main/pandas-blog/pandas-EDA.ipynb">Pandas Blog GitHub repository</a>.</p> <h3 id="preprocessing">Preprocessing</h3> <p>Pre-processing of the data includes cleaning and adding metadata. We will add useful metadata to our dataset.</p> <p>We start with the necessary imports and reading the csv file to a Pandas data frame:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span> <span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span> <span class="n">os_scan_data</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">"../data/OS_Scan_dataset.csv"</span><span class="p">)</span> <span class="n">os_scan_data</span> </code></pre></div></div> <p>That will print a subset of the data since the file is too large:</p> <p><img src="../../../static/images/blog_posts/data-analysis-p2/OS_Scan_dataset.png" alt="" /></p> <p>For more information about Pandas data frames, please check the <a href="https://blog.networktocode.com/post/introduction-to-pandas-for-network-development/">Intro to Pandas</a> blog post.</p> <p>Then, we will create metadata timestamp objects using the <code class="language-plaintext highlighter-rouge">to_datetime</code> function:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">datetime</span> <span class="n">timestamps</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">to_datetime</span><span class="p">(</span><span class="n">os_scan_data</span><span class="p">[</span><span class="s">"Time"</span><span class="p">],</span> <span class="nb">format</span><span class="o">=</span><span class="s">'%Y-%m-%d %H:%M:%S.%f'</span><span class="p">)</span> <span class="n">os_scan_data</span><span class="p">[</span><span class="s">"Time"</span><span class="p">]</span> <span class="o">=</span> <span class="n">timestamps</span> <span class="k">print</span><span class="p">(</span><span class="s">"Timestamps"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">timestamps</span><span class="p">)</span> </code></pre></div></div> <p>The timestamps are shown below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Timestamps</span> <span class="mi">0</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">08</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">12.597437</span> <span class="mi">1</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">08</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">12.597474</span> <span class="mi">2</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">08</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">12.597553</span> <span class="mi">3</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">08</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">12.597558</span> <span class="mi">4</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">08</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">12.597679</span> <span class="p">...</span> <span class="mi">1697846</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">09</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mf">25.354194</span> <span class="mi">1697847</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">09</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mf">25.354321</span> <span class="mi">1697848</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">09</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mf">25.354341</span> <span class="mi">1697849</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">09</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mf">25.354358</span> <span class="mi">1697850</span> <span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">07</span> <span class="mi">09</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mf">25.354493</span> <span class="n">Name</span><span class="p">:</span> <span class="n">Time</span><span class="p">,</span> <span class="n">Length</span><span class="p">:</span> <span class="mi">1697851</span><span class="p">,</span> <span class="n">dtype</span><span class="p">:</span> <span class="n">datetime64</span><span class="p">[</span><span class="n">ns</span><span class="p">]</span> </code></pre></div></div> <p>Finally, we will calculate interesting derivative data, such as the packet interarrivals. To this end, we will use the <code class="language-plaintext highlighter-rouge">numpy</code> function <code class="language-plaintext highlighter-rouge">np.diff</code> that takes as input a column of numbers and subtracts its rows in pairs:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">interarrival_times</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">diff</span><span class="p">(</span><span class="n">timestamps</span><span class="p">)</span> <span class="n">interarrival_times</span> </code></pre></div></div> <p>The packet interarrival values are printed below:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>array([ 37000, 79000, 5000, ..., 20000, 17000, 135000], dtype='timedelta64[ns]') </code></pre></div></div> <p>We append the array to the <code class="language-plaintext highlighter-rouge">os_scan_data</code> type, casting it to int, and print the columns of the dataset to verify that the <code class="language-plaintext highlighter-rouge">Interarrivals</code> column has been appended:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">interarrival_times</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">interarrival_times</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="n">os_scan_data</span><span class="p">[</span><span class="s">"Interarrivals"</span><span class="p">]</span> <span class="o">=</span> <span class="n">interarrival_times</span><span class="p">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span> <span class="n">os_scan_data</span><span class="p">.</span><span class="n">columns</span> </code></pre></div></div> <p>Below are the column names of our data after the preprocessing:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Index(['No.', 'Time', 'Source', 'Destination', 'Protocol', 'Length', 'Info', 'Src Port', 'Dst Port', 'Interarrivals'], dtype='object') </code></pre></div></div> <p>Now we are ready to create pretty graphs!</p> <h3 id="graphical-analysis">Graphical Analysis</h3> <p>In this section, we will focus on two graphical techniques from EDA: histograms and scatter plots. We will demonstrate how to combine the information with <code class="language-plaintext highlighter-rouge">jointplots</code> to analyze black-box datasets.</p> <h4 id="histogram">Histogram</h4> <p>The first graph that we will make may not be pretty, however it demonstrates the value and flexibility of Pandas and graphical analysis for data exploration:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">os_scan_data</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">column</span><span class="o">=</span><span class="p">[</span><span class="s">"Length"</span><span class="p">,</span> <span class="s">"Interarrivals"</span><span class="p">,</span> <span class="s">"Src Port"</span><span class="p">,</span> <span class="s">"Dst Port"</span><span class="p">])</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/data-analysis-p2/histograms.png" alt="" /></p> <p>With a single line of code and the power of Pandas data frames, we already have a set of meaningful plots. The histogram offers a graphical summary of the distribution of a single variable dataset. In the above histograms, we see how the values of <code class="language-plaintext highlighter-rouge">Length</code>, <code class="language-plaintext highlighter-rouge">Interarrivals</code>, <code class="language-plaintext highlighter-rouge">Src Port</code>, and <code class="language-plaintext highlighter-rouge">Dst Port</code> are distributed, i.e., spread, into a continuous interval of values.</p> <p>Histograms offer an insight to the shape of our data and they can be fine tuned to give us a better point of view. The main “ingredient” of the histogram is a bin; a bin corresponds to the bars that you see in the graphs above and its height indicates the number of elements that fall within a range of values. The default size of bins in the data frame <code class="language-plaintext highlighter-rouge">hist</code> <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.hist.html#pandas.DataFrame.hist">function</a> is <code class="language-plaintext highlighter-rouge">10</code>. For example, a bin of size (width) <code class="language-plaintext highlighter-rouge">10</code> and height <code class="language-plaintext highlighter-rouge">1000</code> indicates that there are <code class="language-plaintext highlighter-rouge">1000</code> values <code class="language-plaintext highlighter-rouge">x</code> within the range: <code class="language-plaintext highlighter-rouge">0 &lt;= x &lt; 10</code>. Modifying the bin size is a powerful technique to get additional granularity or a “big picture” view of the data distribution:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">os_scan_data</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">column</span><span class="o">=</span><span class="s">'Length'</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span> <span class="n">os_scan_data</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">column</span><span class="o">=</span><span class="s">'Length'</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span> </code></pre></div></div> <table> <tbody> <tr> <td><img src="../../../static/images/blog_posts/data-analysis-p2/hist-length-20.png" alt="" /></td> <td><img src="../../../static/images/blog_posts/data-analysis-p2/hist-length-100.png" alt="" /></td> </tr> </tbody> </table> <p>There is a whole science in how to fine-tune a histogram’s bin size. A good rule of thumb is that if you have dense data, a large size will give you a good “bird’s-eye view”. In the case of packet lengths, we have sparse of data. Therefore, the smaller bin helps us distinguish the data shape.</p> <h4 id="scatter-plot">Scatter Plot</h4> <p>A scatter plot is another common graphical tool of EDA. Using a scatter plot, we are plotting two variables against each other with the goal of correlating their values:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">os_scan_data</span><span class="p">.</span><span class="n">plot</span><span class="p">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s">'Interarrivals'</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="s">'Src Port'</span><span class="p">)</span> <span class="n">os_scan_data</span><span class="p">.</span><span class="n">plot</span><span class="p">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s">'Interarrivals'</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="s">'Dst Port'</span><span class="p">)</span> </code></pre></div></div> <table> <tbody> <tr> <td><img src="../../../static/images/blog_posts/data-analysis-p2/scatter-src.png" alt="" /></td> <td><img src="../../../static/images/blog_posts/data-analysis-p2/scatter-dst.png" alt="" /></td> </tr> </tbody> </table> <p>The story narrated by these two graphs is that packet interarrival values to source ports have a wider spread, i.e. <code class="language-plaintext highlighter-rouge">0..2 x 10^7</code>, whereas for destination ports these values have half the spread. That may point to slow response or a high speed scan, such as an OS scan! Part of the story is a high usage of low source and destination port numbers. This may point to OS services running on these ports, that are targeted on a wide spread of intervals.</p> <h4 id="joint-plots">Joint Plots</h4> <p>Now let’s combine the scatter and histogram plots for additional insight into our data. We will use an additional plotting <a href="https://seaborn.pydata.org/index.html">package</a>, <code class="language-plaintext highlighter-rouge">seaborn</code>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span> <span class="kn">import</span> <span class="nn">seaborn</span> <span class="k">as</span> <span class="n">sns</span> <span class="n">short_interarrivals</span> <span class="o">=</span> <span class="n">os_scan_data</span><span class="p">[(</span><span class="n">os_scan_data</span><span class="p">[</span><span class="s">'Interarrivals'</span><span class="p">]</span> <span class="o">&lt;</span> <span class="mi">10000</span><span class="p">)</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">os_scan_data</span><span class="p">[</span><span class="s">'Interarrivals'</span><span class="p">]</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)]</span> <span class="n">sns</span><span class="p">.</span><span class="n">jointplot</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s">'Interarrivals'</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="s">'Dst Port'</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="s">'hex'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">short_interarrivals</span><span class="p">)</span> <span class="n">sns</span><span class="p">.</span><span class="n">jointplot</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s">'Interarrivals'</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="s">'Dst Port'</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="s">'kde'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">short_interarrivals</span><span class="p">)</span> <span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span> </code></pre></div></div> <table> <tbody> <tr> <td><img src="../../../static/images/blog_posts/data-analysis-p2/seaborn-hex.png" alt="" /></td> <td><img src="../../../static/images/blog_posts/data-analysis-p2/seaborn-kde.png" alt="" /></td> </tr> </tbody> </table> <p>Note that we used the power of Pandas data frame to define a new frame <code class="language-plaintext highlighter-rouge">short_intervals</code>, where we take interarrivals that are less than 10K nanoseconds. The <code class="language-plaintext highlighter-rouge">hex</code> type plot resembles a scatter plot with histograms on the sides. The color coding of the data points indicates higher concentration of values in this specific area. The <code class="language-plaintext highlighter-rouge">kde</code> (Kernel Distribution Estimate) gives a distribution similar to a histogram, however the centralizing values, i.e., kernels, are visualized as well. The three distinct parts of the graph in <code class="language-plaintext highlighter-rouge">kde</code> will be described with three different mathematical distributions.</p> <h3 id="summary-statistics">Summary Statistics</h3> <p>Summary statistics as part of EDA are extremely useful when dealing with a large set of data:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">short_interarrivals</span><span class="p">.</span><span class="n">describe</span><span class="p">()</span> </code></pre></div></div> <p><img src="../../../static/images/blog_posts/data-analysis-p2/summary-stats.png" alt="" /></p> <p>With a single line of code, the <code class="language-plaintext highlighter-rouge">describe</code> Pandas function gives us several statistics such as percentiles, min, max values, etc. These statistics can lead to distribution fitting and additional insights into the data.</p> <h4 id="autocorrelation">Autocorrelation</h4> <p>Finally, autocorrelation calculations show how much the values within a series, i.e., the length or interarrival values, are related:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">length_series</span> <span class="o">=</span> <span class="n">os_scan_data</span><span class="p">[</span><span class="s">"Length"</span><span class="p">]</span> <span class="n">length_series</span><span class="p">.</span><span class="n">autocorr</span><span class="p">()</span> <span class="mf">0.3938818297281779</span> <span class="n">interarrival_series</span> <span class="o">=</span> <span class="n">os_scan_data</span><span class="p">[</span><span class="s">"Interarrivals"</span><span class="p">]</span> <span class="n">interarrival_series</span><span class="p">.</span><span class="n">autocorr</span><span class="p">()</span> <span class="o">-</span><span class="mf">0.031230988268827732</span> </code></pre></div></div> <p>In this case the packet lengths are positively correlated, which means that if a value is above average, the next value will likely be above average. Negative autocorrelation such as the one that is observed for packet interarrivals, means that if an interarrival is above average, the next interarrival will likely be below average. This is a powerful metric for predictions.</p> <h2 id="recap">Recap</h2> <p>We have reviewed how to use EDA techniques to extract useful information from black-box data. This part of the series data analytics for Network Engineers, offers a deeper understanding of the power of the Pandas library and the statistical techniques that you can implement with it. In the last part of the series, we will review some predictive models. Stay tuned!</p> <p>-Xenia</p> <h2 id="resources">Resources</h2> <ul> <li><a href="https://github.com/mundruid/pandas-blog">GitHub repo</a> with code examples.</li> <li>Part 1: <a href="https://blog.networktocode.com/post/introduction-to-pandas-for-network-development/">Introduction to Pandas for Network Development</a></li> <li><a href="https://blog.networktocode.com/post/jupyter-notebooks-for-development/">Jupyter Notebooks for Development</a></li> </ul>Xenia MountrouidouData analytics is an important skill for every engineer and even more the Network Engineer that goes through large amounts of data for troubleshooting. Networking companies have been moving towards data science integrations for appliances and software. The Arista EOS Network Data Lake is a characteristic example where Artificial Intelligence and Machine Learning are used to analyze data from different resources and lead to actionable decisions.How to Write Better Python Tests for Network Programming2022-01-11T00:00:00+00:002022-01-11T00:00:00+00:00https://blog.networktocode.com/post/how-to-write-better-python-tests-for-network-programming<p>In this blog post, I will share with you a simple technique that helped me a lot in writing better, testable code: writing tests in parallel with developing the code.</p> <h2 id="why-is-it-worth-having-tests">Why Is It Worth Having Tests?</h2> <p>Have you heard any of these?</p> <ul> <li><em>Writing tests slows down development. I will write tests when the code is ready.</em></li> <li><em>It may still change; if I write tests now I will have to rewrite them, so I will write tests when the code is ready.</em></li> </ul> <p>I have heard it countless times, and also said it myself. Today I think it is one of the most common mistakes to leave tests for later. It usually means that tests are not as good as they could be or there are no tests at all <em>due to other priorities</em>. Furthermore, if you expect your code to change that is actually a good argument to have tests. When you expect changes, you know that you will eventually have to retest. Perhaps you will have to amend your tests, but when some of your tests fail after the change, you get the extra verification that it’s only related to the change. Lack of decent tests results in technical debt. And like any debt, sooner or later you will have to pay it off. It usually happens when you go back to your code after a while to change/fix something, and all that time you could spend writing tests you will probably spend on manually retesting your code after changing or fixing something. If you still remember how you tested it before, this may be manageable; if not, you will spend even more time on it. You can even skip testing and rely on the grace of the gods that it will work well. But you may avoid all of this if you change just one thing!</p> <h2 id="how-do-you-run-your-code">How Do You Run Your Code?</h2> <p><code class="language-plaintext highlighter-rouge">python &lt;your_file&gt;.py</code> Right? OK, time for the pro tip!</p> <p>What if you avoid running code directly and run it with tests instead?</p> <h2 id="development-through-tests">Development Through Tests</h2> <p>When developing code, we write functions, classes, methods. And we run them to test whether they give us what we expect. Running your code for the first time is the right time to develop tests! All you need to do is just run your code with <code class="language-plaintext highlighter-rouge">pytest</code> instead of running it directly; capture outputs which you normally check with <code class="language-plaintext highlighter-rouge">print()</code>; and gradually build your tests as you develop your code.</p> <p>Let’s get our hands dirty by creating some practical examples. This is our project structure:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── main.py └── tests ├── __init__.py └── test_main.py </code></pre></div></div> <p>Create our first function in <code class="language-plaintext highlighter-rouge">main.py</code>, something simple.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># main.py </span><span class="k">def</span> <span class="nf">simple_math_function</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span> <span class="s">"""Sum arguments"""</span> <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span> <span class="n">total</span> <span class="o">+=</span> <span class="n">arg</span> <span class="k">return</span> <span class="n">total</span> </code></pre></div></div> <p>Now we should test our function to check whether we get what we expect. But instead of running <code class="language-plaintext highlighter-rouge">python main.py</code>, we create a test in <code class="language-plaintext highlighter-rouge">tests/test_main.py</code> and we run <code class="language-plaintext highlighter-rouge">pytest -s</code>. Remember the <code class="language-plaintext highlighter-rouge">-s</code> option, as it gives all <code class="language-plaintext highlighter-rouge">print()</code> outputs on-screen. We use <code class="language-plaintext highlighter-rouge">print</code> in the test, but you can use it anywhere in your code. Now we just want to capture our print the same way we would by running <code class="language-plaintext highlighter-rouge">python main.py</code> and calling our function there.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_simple_math_function</span><span class="p">():</span> <span class="n">o</span> <span class="o">=</span> <span class="n">main</span><span class="p">.</span><span class="n">simple_math_function</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">o</span><span class="p">)</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -s ============================== test session starts ============================ platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 1 item tests/test_main.py 15 . ============================== 1 passed in 0.01s =============================== </code></pre></div></div> <p>I usually use <code class="language-plaintext highlighter-rouge">-k</code> option to point to a specific test. This is convenient when you already have many tests, and you want to work on one. Let’s run tests again, limiting them to only the test we work on.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -s -k simple_math_function ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 1 item tests/test_main.py 15 . ============================== 1 passed in 0.01s =============================== </code></pre></div></div> <p>Our output is <code class="language-plaintext highlighter-rouge">15</code>, and it is indeed the sum of all the arguments we passed to our function. Now we can just replace <code class="language-plaintext highlighter-rouge">print</code> with <code class="language-plaintext highlighter-rouge">assert</code> and we now have a test that compares the function call result with our previously captured expected result. We have our first test completed, which will remain and will be executed automatically whenever we run our tests in the future.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_simple_math_function</span><span class="p">():</span> <span class="k">assert</span> <span class="n">main</span><span class="p">.</span><span class="n">simple_math_function</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="o">==</span> <span class="mi">15</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -s -v -k our_simple_function ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 1 item tests/test_main.py::test_our_simple_function PASSED ============================== 1 passed in 0.02s =============================== </code></pre></div></div> <p>Note <code class="language-plaintext highlighter-rouge">-v</code> option, which gives more verbose output. Let’s make one more function and test.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># main.py </span><span class="k">def</span> <span class="nf">simple_hello</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">return</span> <span class="sa">f</span><span class="s">"Hello dear </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s">!"</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_simple_hello</span><span class="p">():</span> <span class="k">print</span><span class="p">(</span><span class="n">main</span><span class="p">.</span><span class="n">simple_hello</span><span class="p">(</span><span class="s">"Guest"</span><span class="p">))</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k simple_hello ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 2 items / 1 deselected / 1 selected tests/test_main.py::test_simple_hello Hello dear Guest! PASSED ========================== 1 passed, 1 deselected in 0.02s ======================= </code></pre></div></div> <p>Again we modify <code class="language-plaintext highlighter-rouge">print</code> to <code class="language-plaintext highlighter-rouge">assert</code>, and we add the expected result and run the test again.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_simple_hello</span><span class="p">():</span> <span class="k">assert</span> <span class="n">main</span><span class="p">.</span><span class="n">simple_hello</span><span class="p">(</span><span class="s">"Guest"</span><span class="p">)</span> <span class="o">==</span> <span class="s">"Hello dear Guest!"</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k simple_hello ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 2 items / 1 deselected / 1 selected tests/test_main.py::test_simple_hello PASSED ========================= 1 passed, 1 deselected in 0.03s ======================= </code></pre></div></div> <p>As you see, the effort is comparable to typical testing with <code class="language-plaintext highlighter-rouge">print</code>, but with a little more effort, we have unit tests that will remain after we remove print statements. This is a huge benefit for the future and for anyone else who will work with our code.</p> <h2 id="practice-makes-perfect">Practice Makes Perfect</h2> <p>Let’s develop something more practical from the networking world. We will use <code class="language-plaintext highlighter-rouge">netmiko</code> to get software version from a device, and we develop that through tests.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># main.py </span><span class="kn">from</span> <span class="nn">netmiko</span> <span class="kn">import</span> <span class="n">ConnectHandler</span> <span class="k">def</span> <span class="nf">get_running_version</span><span class="p">(</span><span class="n">driver</span><span class="p">,</span> <span class="n">host</span><span class="p">,</span> <span class="n">username</span><span class="o">=</span><span class="s">"admin"</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="s">"admin"</span><span class="p">):</span> <span class="k">with</span> <span class="n">ConnectHandler</span><span class="p">(</span> <span class="n">device_type</span><span class="o">=</span><span class="n">driver</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">username</span><span class="o">=</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="n">password</span> <span class="p">)</span> <span class="k">as</span> <span class="n">device</span><span class="p">:</span> <span class="n">version</span> <span class="o">=</span> <span class="n">device</span><span class="p">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">"show version"</span><span class="p">,</span> <span class="n">use_textfsm</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">return</span> <span class="n">version</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_get_running_version</span><span class="p">():</span> <span class="n">version</span> <span class="o">=</span> <span class="n">main</span><span class="p">.</span><span class="n">get_running_version</span><span class="p">(</span><span class="s">"cisco_ios"</span><span class="p">,</span> <span class="s">"10.1.1.1"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">version</span><span class="p">)</span> </code></pre></div></div> <p>Let’s run to see what we get from the device.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k get_running_version ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 3 items / 2 deselected / 1 selected tests/test_main.py::test_get_running_version [{'version': '15.7(3)M5', 'rommon': 'System', 'hostname': 'LANRTR01', 'uptime': '1 year, 42 weeks, 4 days, 1 hour, 18 minutes', 'uptime _years': '1', 'uptime_weeks': '42', 'uptime_days': '4', 'uptime_hours': '1', 'uptime_minutes': '18', 'reload_reason': 'Reload Command', 'running_image': 'c2951-universalk9-mz.SPA.157 -3.M5.bin', 'hardware': ['CISCO2951/K9'], 'serial': ['FGL2014508V'], 'config_register': '0x2102', 'mac': [], 'restarted': '10:48:48 GMT Fri Mar 6 2020'}] PASSED ======================== 1 passed, 2 deselected in 6.01s ========================= </code></pre></div></div> <p>We need index <code class="language-plaintext highlighter-rouge">0</code> and the <code class="language-plaintext highlighter-rouge">version</code> key. We modify the return in our function in <code class="language-plaintext highlighter-rouge">main.py</code> and run the test again.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># main.py </span><span class="k">def</span> <span class="nf">get_running_version</span><span class="p">(</span><span class="n">driver</span><span class="p">,</span> <span class="n">host</span><span class="p">,</span> <span class="n">username</span><span class="o">=</span><span class="s">"admin"</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="s">"admin"</span><span class="p">):</span> <span class="k">with</span> <span class="n">ConnectHandler</span><span class="p">(</span> <span class="n">device_type</span><span class="o">=</span><span class="n">driver</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">username</span><span class="o">=</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="n">password</span> <span class="p">)</span> <span class="k">as</span> <span class="n">device</span><span class="p">:</span> <span class="n">version</span> <span class="o">=</span> <span class="n">device</span><span class="p">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">"show version"</span><span class="p">,</span> <span class="n">use_textfsm</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">return</span> <span class="n">version</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s">"version"</span><span class="p">]</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k get_running_version ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 3 items / 2 deselected / 1 selected tests/test_main.py::test_get_running_version 15.7(3)M5 PASSED ========================= 1 passed, 2 deselected in 9.02s ======================= </code></pre></div></div> <p>Now we can modify our test: remove <code class="language-plaintext highlighter-rouge">print</code> and add <code class="language-plaintext highlighter-rouge">assert</code> and enter the returned value as the expected value, then we run the test again.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_get_running_version</span><span class="p">():</span> <span class="n">version</span> <span class="o">=</span> <span class="n">main</span><span class="p">.</span><span class="n">get_running_version</span><span class="p">(</span><span class="s">"cisco_ios"</span><span class="p">,</span> <span class="s">"10.1.1.1"</span><span class="p">)</span> <span class="k">assert</span> <span class="n">version</span> <span class="o">==</span> <span class="s">"15.7(3)M5"</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k get_running_version ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 3 items / 2 deselected / 1 selected tests/test_main.py::test_get_running_version PASSED ======================== 1 passed, 2 deselected in 8.01s ========================= </code></pre></div></div> <p>Our test works fine, but it takes 8 sec to complete because we still connect to the real device. We need to mock up netmiko output. Under <code class="language-plaintext highlighter-rouge">tests/conftest.py</code>, we create <code class="language-plaintext highlighter-rouge">FakeDevice</code> class, where we overwrite netmiko <code class="language-plaintext highlighter-rouge">send_command</code> method, which we use to get structured output of <code class="language-plaintext highlighter-rouge">show version</code>, and we return the same output that we collected from the device with print. Because we call ConnectHandler with context manager, we also need to implement <code class="language-plaintext highlighter-rouge">__enter__</code> and <code class="language-plaintext highlighter-rouge">__exit__</code> methods. Next we create <code class="language-plaintext highlighter-rouge">mock_netmiko</code> fixture where we use pytest <code class="language-plaintext highlighter-rouge">monkeypatch</code> to patch <code class="language-plaintext highlighter-rouge">ConnectHandler</code> in our <code class="language-plaintext highlighter-rouge">main.py</code> module. This fixture we use as an argument in our test function. You can <a href="https://docs.pytest.org/en/latest/how-to/monkeypatch.html">read more</a> on how to mock/monkeypatch in pytest documentation.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># tests/conftest.py </span><span class="kn">import</span> <span class="nn">pytest</span> <span class="kn">import</span> <span class="nn">main</span> <span class="k">class</span> <span class="nc">FakeDevice</span><span class="p">:</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">pass</span> <span class="k">def</span> <span class="nf">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span> <span class="k">def</span> <span class="nf">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="k">pass</span> <span class="k">def</span> <span class="nf">send_command</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">return</span> <span class="p">[</span> <span class="p">{</span> <span class="s">'version'</span><span class="p">:</span> <span class="s">'15.7(3)M5'</span><span class="p">,</span> <span class="s">'rommon'</span><span class="p">:</span> <span class="s">'System'</span><span class="p">,</span> <span class="s">'hostname'</span><span class="p">:</span> <span class="s">'LANRTR01'</span><span class="p">,</span> <span class="s">'uptime'</span><span class="p">:</span> <span class="s">'1 year, 42 weeks, 4 days, 1 hour, 18 minutes'</span><span class="p">,</span> <span class="s">'uptime_years'</span><span class="p">:</span> <span class="s">'1'</span><span class="p">,</span> <span class="s">'uptime_weeks'</span><span class="p">:</span> <span class="s">'42'</span><span class="p">,</span> <span class="s">'uptime_days'</span><span class="p">:</span> <span class="s">'4'</span><span class="p">,</span> <span class="s">'uptime_hours'</span><span class="p">:</span> <span class="s">'1'</span><span class="p">,</span> <span class="s">'uptime_minutes'</span><span class="p">:</span> <span class="s">'18'</span><span class="p">,</span> <span class="s">'reload_reason'</span><span class="p">:</span> <span class="s">'Reload Command'</span><span class="p">,</span> <span class="s">'running_image'</span><span class="p">:</span> <span class="s">'c2951-universalk9-mz.SPA.157-3.M5.bin'</span><span class="p">,</span> <span class="s">'hardware'</span><span class="p">:</span> <span class="p">[</span><span class="s">'CISCO2951/K9'</span><span class="p">],</span> <span class="s">'serial'</span><span class="p">:</span> <span class="p">[</span><span class="s">'FGL2014508V'</span><span class="p">],</span> <span class="s">'config_register'</span><span class="p">:</span> <span class="s">'0x2102'</span><span class="p">,</span> <span class="s">'mac'</span><span class="p">:</span> <span class="p">[],</span> <span class="s">'restarted'</span><span class="p">:</span> <span class="s">'10:48:48 GMT Fri Mar 6 2020'</span> <span class="p">}</span> <span class="p">]</span> <span class="o">@</span><span class="n">pytest</span><span class="p">.</span><span class="n">fixture</span><span class="p">()</span> <span class="k">def</span> <span class="nf">mock_netmiko</span><span class="p">(</span><span class="n">monkeypatch</span><span class="p">):</span> <span class="s">"""Mock netmiko."""</span> <span class="n">monkeypatch</span><span class="p">.</span><span class="nb">setattr</span><span class="p">(</span><span class="n">main</span><span class="p">,</span> <span class="s">"ConnectHandler"</span><span class="p">,</span> <span class="n">FakeDevice</span><span class="p">)</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># /tests/test_main.py </span><span class="kn">import</span> <span class="nn">main</span> <span class="k">def</span> <span class="nf">test_get_running_version</span><span class="p">(</span><span class="n">mock_netmiko</span><span class="p">):</span> <span class="n">version</span> <span class="o">=</span> <span class="n">main</span><span class="p">.</span><span class="n">get_running_version</span><span class="p">(</span><span class="s">"cisco_ios"</span><span class="p">,</span> <span class="s">"10.1.1.1"</span><span class="p">)</span> <span class="k">assert</span> <span class="n">version</span> <span class="o">==</span> <span class="s">"15.7(3)M5"</span> </code></pre></div></div> <p>We run the test again.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pytest -sv -k get_running_version ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/patryk/projects/pytest_mock_blog, configfile: pytest.ini collected 3 items / 2 deselected / 1 selected tests/test_main.py::test_get_running_version PASSED =========================== 1 passed, 2 deselected in 0.02s ===================== </code></pre></div></div> <p>This time it took only 0.02 sec to execute the test because we used mock and did not connect to the device anymore.</p> <h2 id="more-on-developing-tests">More on Developing Tests</h2> <p>Check out <a href="https://github.com/pszulczewski/netmiko_sandbox">Netmiko Sandbox</a>, where you can get more practice with structured command output from multiple vendor devices—all available as code, so you don’t even have to run any device! You can also easily collect command outputs for your mocks.</p> <p>Also check out Adam’s awesome series of blog posts on pytest in the networking world, where Adam shares practical fundamentals of testing. <a href="https://blog.networktocode.com/post/pytest-in-the-networking-world">Part 1</a> <a href="https://blog.networktocode.com/post/pytest-in-the-netwoking-world-part-2">Part 2</a> <a href="https://blog.networktocode.com/post/pytest-in-the-netwoking-world-part-3">Part 3</a> Pay attention to test parametrization and consider how we could extend our first two tests with more parameters.</p> <h2 id="conclusion">Conclusion</h2> <p>It may seem like Test Driven Development, but is it really TDD? Well, TDD principles say that a test is developed first, before the actual code that makes the test pass. In this approach code and tests are developed in parallel, so formally it doesn’t strictly follow TDD principles. I would put this in between TDD and the typical code development followed by tests.</p> <p>The presented approach to testing requires you to change your habits of how you run your code during development, but it has several significant advantages:</p> <ul> <li>tests are developed in parallel with code, <em>Will do it later</em> is avoided</li> <li>manual tests are input to automated tests, work on manual tests done once can be automatically executed later</li> <li>better code quality, developed code is testable, you will not be able to develop tests for untestable code</li> <li>increased test coverage right from the beginning as opposed to tests developed later</li> <li>greater confidence after implementing changes/fixes as all tests can be performed instantaneously and automatically</li> </ul> <p>-Patryk</p>Patryk SzulczewskiIn this blog post, I will share with you a simple technique that helped me a lot in writing better, testable code: writing tests in parallel with developing the code.Advanced Options for Building a Nautobot SSoT App2022-01-06T00:00:00+00:002022-01-06T00:00:00+00:00https://blog.networktocode.com/post/advanced-ssot-app<p>In the <a href="https://blog.networktocode.com/post/building-a-nautobot-ssot-app/">first part</a> of this series, we reviewed the building blocks of an SSoT app for Nautobot. We reviewed the design of DiffSyncModel classes, the CRUD methods on those classes, building your System of Record adapters to fill those models, and finally the Nautobot Job that executes the synchronization of data between your Systems of Record. In this second half, we’ll review advanced options available to you when architecting an SSoT app like controlling the order of processing for your data and handling special requirements for object deletion.</p> <blockquote> <p>Please note: it is expected that you’ve read the <a href="https://blog.networktocode.com/post/nautobot-ssot-plugin/">Nautobot Plugin: Single Source of Truth (SSoT)</a> and <a href="https://blog.networktocode.com/post/building-a-nautobot-ssot-app/">Building a Nautobot SSoT App</a> posts and understand the framework terminology, such as Data Source and Data Target.</p> </blockquote> <p>In the designing of your SSoT application you might find yourself in a situation where you want to define the processing order of your SoR objects. The standard method of processing a set of objects in the DiffSync process can’t be guaranteed but is typically a simple first in, first out queue defined by the order the objects were presented by your adapters. To change this behavior you can extend the <code class="language-plaintext highlighter-rouge">Diff</code> class itself and define the processing order. One option might be to process each class of your objects alphabetically, as shown in the example below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span> <span class="kn">from</span> <span class="nn">diffsync.diff</span> <span class="kn">import</span> <span class="n">Diff</span> <span class="k">class</span> <span class="nc">CustomOrderingDiff</span><span class="p">(</span><span class="n">Diff</span><span class="p">):</span> <span class="s">"""Alternate diff class to list children in alphabetical order, except devices to be ordered by CRUD action."""</span> <span class="o">@</span><span class="nb">classmethod</span> <span class="k">def</span> <span class="nf">order_children_default</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">children</span><span class="p">):</span> <span class="s">"""Simple diff to return all children in alphabetical order."""</span> <span class="k">for</span> <span class="n">child_name</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">children</span><span class="p">.</span><span class="n">items</span><span class="p">()):</span> <span class="k">yield</span> <span class="n">children</span><span class="p">[</span><span class="n">child_name</span><span class="p">]</span> </code></pre></div></div> <p>In some cases you wish to have this done for a single object type, like Devices. This can be done by having a method in your custom <code class="language-plaintext highlighter-rouge">Diff</code> class named after the type of the object in the pattern <code class="language-plaintext highlighter-rouge">order_children_&lt;type&gt;</code>. It will utilize the <code class="language-plaintext highlighter-rouge">order_children_default</code> method for any other object classes that haven’t been explicitly defined. This option also allows you to control the order of CRUD operations that happen on a particular object, as shown in the example below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="o">@</span><span class="nb">classmethod</span> <span class="k">def</span> <span class="nf">order_children_device</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">children</span><span class="p">):</span> <span class="s">"""Return a list of device sorted by CRUD action, starting with deletion, then create, and update, along with being in alphabetical order."""</span> <span class="n">children_by_type</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> <span class="c1"># Organize the children's name by action create, update, or delete </span> <span class="k">for</span> <span class="n">child_name</span><span class="p">,</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">children</span><span class="p">.</span><span class="n">items</span><span class="p">():</span> <span class="n">action</span> <span class="o">=</span> <span class="n">child</span><span class="p">.</span><span class="n">action</span> <span class="ow">or</span> <span class="s">"skip"</span> <span class="n">children_by_type</span><span class="p">[</span><span class="n">action</span><span class="p">].</span><span class="n">append</span><span class="p">(</span><span class="n">child_name</span><span class="p">)</span> <span class="c1"># Create a global list, organized per action, with deletion first to prevent conflicts </span> <span class="n">sorted_children</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">children_by_type</span><span class="p">[</span><span class="s">"delete"</span><span class="p">])</span> <span class="n">sorted_children</span> <span class="o">+=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">children_by_type</span><span class="p">[</span><span class="s">"create"</span><span class="p">])</span> <span class="n">sorted_children</span> <span class="o">+=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">children_by_type</span><span class="p">[</span><span class="s">"update"</span><span class="p">])</span> <span class="n">sorted_children</span> <span class="o">+=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">children_by_type</span><span class="p">[</span><span class="s">"skip"</span><span class="p">])</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">sorted_children</span><span class="p">:</span> <span class="k">yield</span> <span class="n">children</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> </code></pre></div></div> <p>Once you’ve defined your custom <code class="language-plaintext highlighter-rouge">Diff</code> ordering class you simply need to pass it to the appropriate <code class="language-plaintext highlighter-rouge">diff_from</code>/<code class="language-plaintext highlighter-rouge">diff_to</code> or <code class="language-plaintext highlighter-rouge">sync_from</code>/<code class="language-plaintext highlighter-rouge">sync_to</code> methods, as shown below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSyncFlags</span> <span class="kn">from</span> <span class="nn">diffsync.exceptions</span> <span class="kn">import</span> <span class="n">ObjectNotCreated</span> <span class="k">def</span> <span class="nf">sync_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="s">"""SSoT synchronization from Device42 to Nautobot."""</span> <span class="n">client</span> <span class="o">=</span> <span class="n">Device42API</span><span class="p">()</span> <span class="n">d42_adapter</span> <span class="o">=</span> <span class="n">Device42Adapter</span><span class="p">(</span><span class="n">job</span><span class="o">=</span><span class="bp">self</span><span class="p">,</span> <span class="n">sync</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">,</span> <span class="n">client</span><span class="o">=</span><span class="n">client</span><span class="p">)</span> <span class="n">d42_adapter</span><span class="p">.</span><span class="n">load</span><span class="p">()</span> <span class="n">nb_adapter</span> <span class="o">=</span> <span class="n">NautobotAdapter</span><span class="p">(</span><span class="n">job</span><span class="o">=</span><span class="bp">self</span><span class="p">,</span> <span class="n">sync</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">)</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">load</span><span class="p">()</span> <span class="n">diff</span> <span class="o">=</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">diff_from</span><span class="p">(</span><span class="n">d42_adapter</span><span class="p">,</span> <span class="n">diff_class</span><span class="o">=</span><span class="n">CustomOrderingDiff</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="p">.</span><span class="n">kwargs</span><span class="p">[</span><span class="s">"dry_run"</span><span class="p">]:</span> <span class="k">try</span><span class="p">:</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">sync_from</span><span class="p">(</span><span class="n">d42_adapter</span><span class="p">,</span> <span class="n">diff_class</span><span class="o">=</span><span class="n">CustomOrderingDiff</span><span class="p">)</span> <span class="k">except</span> <span class="n">ObjectNotCreated</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_debug</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="sa">f</span><span class="s">"Unable to create object. </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_success</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="s">"Sync complete."</span><span class="p">)</span> </code></pre></div></div> <p>Custom <code class="language-plaintext highlighter-rouge">Diff</code> classes can come in handy when you need to ensure that an obsolete version of an object has been removed before a newer version being installed to prevent possible conflicts.</p> <p>In addition to controlling the flow of your object processing, you might have a situation where the synchronization fails or you only want to consider objects that exist in one of or both of your Systems of Record. In these cases you would want to utilize a DiffSync <em>Flag</em>. The core DiffSync engine provides two sets of flags, allowing for modifying behavior of DiffSync at either the Global or Model level. As the name implies, <em>global flags</em> apply to all data and while <em>model flags</em> apply to a specific model. A list of the included <em>global flag</em> options (as of DiffSync 1.3) has been provided below:</p> <table> <thead> <tr> <th>Name</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="language-plaintext highlighter-rouge">CONTINUE_ON_FAILURE</code></td> <td>Continue synchronizing even if failures are encountered when syncing individual models.</td> </tr> <tr> <td><code class="language-plaintext highlighter-rouge">SKIP_UNMATCHED_SRC</code></td> <td>Ignore objects that only exist in the source/”from” DiffSync when determining diffs and syncing. If this flag is set, no new objects will be created in the target/”to” DiffSync.</td> </tr> <tr> <td><code class="language-plaintext highlighter-rouge">SKIP_UNMATCHED_DST</code></td> <td>Ignore objects that only exist in the target/”to” DiffSync when determining diffs and syncing. If this flag is set, no objects will be deleted from the target/”to” DiffSync.</td> </tr> <tr> <td><code class="language-plaintext highlighter-rouge">SKIP_UNMATCHED_BOTH</code></td> <td>Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag</td> </tr> <tr> <td><code class="language-plaintext highlighter-rouge">LOG_UNCHANGED_RECORDS</code></td> <td>If this flag is set, a log message will be generated during synchronization for each model, even unchanged ones.</td> </tr> </tbody> </table> <p>Like your custom <code class="language-plaintext highlighter-rouge">Diff</code> ordering class, utilizing the <em>global flags</em> simply requires applying them to the appropriate <code class="language-plaintext highlighter-rouge">diff</code> and <code class="language-plaintext highlighter-rouge">sync</code> methods in your Job, as below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync.enum</span> <span class="kn">import</span> <span class="n">DiffSyncFlags</span> <span class="n">flags</span> <span class="o">=</span> <span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">CONTINUE_ON_FAILURE</span> <span class="n">diff</span> <span class="o">=</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">diff_from</span><span class="p">(</span><span class="n">d42_adapter</span><span class="p">,</span> <span class="n">diff_class</span><span class="o">=</span><span class="n">CustomOrderingDiff</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">flags</span><span class="p">)</span> </code></pre></div></div> <p><em>Model flags</em> are applied to individual <code class="language-plaintext highlighter-rouge">DiffSyncModel</code> instances, for example, you could apply them from the adapter’s <code class="language-plaintext highlighter-rouge">load</code> method, as shown in the example below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span> <span class="kn">from</span> <span class="nn">diffsync.enum</span> <span class="kn">import</span> <span class="n">DiffSyncModelFlags</span> <span class="kn">from</span> <span class="nn">models</span> <span class="kn">import</span> <span class="n">Device</span> <span class="k">class</span> <span class="nc">NSOAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="n">device</span> <span class="o">=</span> <span class="n">Device</span> <span class="k">def</span> <span class="nf">load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="s">"""Load all devices into the adapter and add the flag IGNORE to all non-ACI devices."""</span> <span class="k">for</span> <span class="n">device</span> <span class="ow">in</span> <span class="n">data</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"devices"</span><span class="p">):</span> <span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">device</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">device</span><span class="p">[</span><span class="s">"name"</span><span class="p">])</span> <span class="k">if</span> <span class="s">"ACI"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">device</span><span class="p">[</span><span class="s">"name"</span><span class="p">]:</span> <span class="n">obj</span><span class="p">.</span><span class="n">model_flags</span> <span class="o">=</span> <span class="n">DiffSyncModelFlags</span><span class="p">.</span><span class="n">IGNORE</span> <span class="bp">self</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> </code></pre></div></div> <p>The DiffSync library, as of version 1.3, currently includes two options for <code class="language-plaintext highlighter-rouge">Model Flags</code>, as shown in the table below:</p> <table> <thead> <tr> <th>Name</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="language-plaintext highlighter-rouge">IGNORE</code></td> <td>Do not render diffs containing this model; do not make any changes to this model when synchronizing. Can be used to indicate a model instance that exists but should not be changed by DiffSync.</td> </tr> <tr> <td><code class="language-plaintext highlighter-rouge">SKIP_CHILDREN_ON_DELETE</code></td> <td>When deleting this model, do not recursively delete its children. Can be used for the case where deletion of a model results in the automatic deletion of all its children.</td> </tr> </tbody> </table> <p>Both <em>global flags</em> and <em>model flags</em> are stored as a binary representation. This allows for storage of multiple flags within a single variable and allows for additional flags to be added in the future. Due to the nature of each flag being a different binary value it is necessary to perform a bitwise OR operation when utilizing multiple flags at once. Imagine the scenario where you want to skip objects that don’t exist in either Systems of Record and log all object records regardless of their being changed. You would first need to define one flag and then perform the bitwise OR operation, as shown in the example:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">diffsync.enum</span> <span class="kn">import</span> <span class="n">DiffSyncFlags</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">flags</span> <span class="o">=</span> <span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">SKIP_UNMATCHED_BOTH</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">flags</span> <span class="o">&lt;</span><span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">SKIP_UNMATCHED_BOTH</span><span class="p">:</span> <span class="mi">6</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">bin</span><span class="p">(</span><span class="n">flags</span><span class="p">.</span><span class="n">value</span><span class="p">)</span> <span class="s">'0b110'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">LOG_UNCHANGED_RECORDS</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">flags</span> <span class="o">&lt;</span><span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">LOG_UNCHANGED_RECORDS</span><span class="o">|</span><span class="n">SKIP_UNMATCHED_BOTH</span><span class="o">|</span><span class="n">SKIP_UNMATCHED_DST</span><span class="o">|</span><span class="n">SKIP_UNMATCHED_SRC</span><span class="p">:</span> <span class="mi">14</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">bin</span><span class="p">(</span><span class="n">flags</span><span class="p">.</span><span class="n">value</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="s">'0b1110'</span> </code></pre></div></div> <p>Now that you’ve defined exactly how you want your SSoT application to handle the data from your Systems of Record you might have a requirement to perform some action on the data once the sync has completed. Luckily, the SSoT app makes this easy by looking for a <code class="language-plaintext highlighter-rouge">sync_complete</code> method in your DataTarget adapter and running it if found. A case where this could be used is one where deletion of objects in your Systems of Record needs to be handled in a specific manner due to inter-object dependence. An example of this would be something like a Site in Nautobot that can’t be deleted until all devices, racks, and other objects in that Site have been deleted or moved. To perform this operation you would need to define your object’s <code class="language-plaintext highlighter-rouge">delete</code> method, as below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="s">"""Delete Site object from Nautobot. Because Site has a direct relationship with many other objects, it can't be deleted before anything else. The self.diffsync.objects_to_delete dictionary stores all objects for deletion and removes them from Nautobot in the correct order. This is used in the Nautobot adapter sync_complete function. """</span> <span class="bp">self</span><span class="p">.</span><span class="n">diffsync</span><span class="p">.</span><span class="n">job</span><span class="p">.</span><span class="n">log_warning</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="sa">f</span><span class="s">"Site </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s"> will be deleted."</span><span class="p">)</span> <span class="nb">super</span><span class="p">().</span><span class="n">delete</span><span class="p">()</span> <span class="n">site</span> <span class="o">=</span> <span class="n">Site</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">uuid</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">diffsync</span><span class="p">.</span><span class="n">objects_to_delete</span><span class="p">[</span><span class="s">"site"</span><span class="p">].</span><span class="n">append</span><span class="p">(</span><span class="n">site</span><span class="p">)</span> <span class="c1"># pylint: disable=protected-access </span> <span class="k">return</span> <span class="bp">self</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">delete</code> method is marking the object as deleted, but instead of deleting it immediately from Nautobot’s database, it is adding it to a list of objects to be removed once the synchronization has completed and the appropriate order of deleting objects can be performed, as shown in the following example:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span> <span class="kn">from</span> <span class="nn">django.db.models</span> <span class="kn">import</span> <span class="n">ProtectedError</span> <span class="k">class</span> <span class="nc">NautobotAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="s">"""Nautobot adapter for DiffSync."""</span> <span class="n">objects_to_delete</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> <span class="k">def</span> <span class="nf">sync_complete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">:</span> <span class="n">DiffSync</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="s">"""Clean up function for DiffSync sync. Once the sync is complete, this function runs deleting any objects from Nautobot that need to be deleted in a specific order. Args: source (DiffSync): DiffSync """</span> <span class="k">for</span> <span class="n">grouping</span> <span class="ow">in</span> <span class="p">(</span> <span class="s">"ipaddr"</span><span class="p">,</span> <span class="s">"subnet"</span><span class="p">,</span> <span class="s">"vrf"</span><span class="p">,</span> <span class="s">"vlan"</span><span class="p">,</span> <span class="s">"cluster"</span><span class="p">,</span> <span class="s">"port"</span><span class="p">,</span> <span class="s">"device"</span><span class="p">,</span> <span class="s">"device_type"</span><span class="p">,</span> <span class="s">"manufacturer"</span><span class="p">,</span> <span class="s">"rack"</span><span class="p">,</span> <span class="s">"site"</span><span class="p">,</span> <span class="c1"># can't delete a site until all of its dependent objects, above, have been deleted </span> <span class="p">):</span> <span class="k">for</span> <span class="n">nautobot_object</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">objects_to_delete</span><span class="p">[</span><span class="n">grouping</span><span class="p">]:</span> <span class="k">try</span><span class="p">:</span> <span class="n">nautobot_object</span><span class="p">.</span><span class="n">delete</span><span class="p">()</span> <span class="k">except</span> <span class="n">ProtectedError</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">job</span><span class="p">.</span><span class="n">log_failure</span><span class="p">(</span><span class="n">obj</span><span class="o">=</span><span class="n">nautobot_object</span><span class="p">,</span> <span class="n">message</span><span class="o">=</span><span class="s">"Deletion failed protected object"</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">objects_to_delete</span><span class="p">[</span><span class="n">grouping</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">return</span> <span class="nb">super</span><span class="p">().</span><span class="n">sync_complete</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> </code></pre></div></div> <p>Just be aware that any changes made to your Systems of Record through the <code class="language-plaintext highlighter-rouge">sync_complete</code> method should be ones that won’t impact the data sets between your Systems of Record. This is essential to minimize unnecessary updates on subsequent runs of your sync Job. An example of this would be performing a DNS query of your devices and creating IP Addresses and interfaces on devices after the sync is complete. Doing so would cause these same IP Addresses and interfaces to be absent in the comparison of your Systems of Record and would thus be deleted and then re-added again after the sync finished. This would cause a repeated cycle of objects being removed and re-added. The best practice is to have any manipulation of your data sets in your Systems of Record performed within your adapters and prior to the diff and sync are performed.</p> <p>Now that you know the basics of designing an SSoT app and have been enlightened to the power of global and model DiffSync flags, custom Diff classes, and the <code class="language-plaintext highlighter-rouge">sync_complete</code> method, the options for designing your Single Source of Truth application are limited only by your imagination. We at Network to Code look forward to seeing what you and the community creates.</p> <p>-Justin</p>Justin DrewIn the first part of this series, we reviewed the building blocks of an SSoT app for Nautobot. We reviewed the design of DiffSyncModel classes, the CRUD methods on those classes, building your System of Record adapters to fill those models, and finally the Nautobot Job that executes the synchronization of data between your Systems of Record. In this second half, we’ll review advanced options available to you when architecting an SSoT app like controlling the order of processing for your data and handling special requirements for object deletion.Working with Webhooks in Nautobot2021-12-23T00:00:00+00:002021-12-23T00:00:00+00:00https://blog.networktocode.com/post/nautobot-webhooks<p>This week’s Nautobot blog deals with <em>webhooks</em>. In basic terms, a webhook is a method for one web application to programmatically provide information to another web app.</p> <p>The use case for webhooks is predicated on <em>events</em>: when a certain event happens, another event should happen in response. If that response is an action by a different app, then a webhook can be sent to notify the app that it needs to take action. The webhook can also carry information to that receiving app, including:</p> <ul> <li>Notification that a specific event happened</li> <li>Information about the event</li> </ul> <p>Let’s illustrate this with a practical use case. This post will describe how to create a webhook in Nautobot that will trigger when a new device is created. The webhook will notify Microsoft Teams to post a message about the event in a channel.</p> <h2 id="getting-a-target-url-ms-teams-example">Getting a Target URL (MS Teams Example)</h2> <p>A webhook needs a <em>target URL</em>: this is a destination endpoint for the webhook to send its information to.</p> <p>This example will use MS Teams, but most of the chat platforms have easy methods for creating incoming webhook endpoints.</p> <p>It’s quite simple to get an incoming webhook URL for MS Teams:</p> <p><strong>1.</strong> In the desired Teams channel, click on the three dots (…) in the top-right corner of the channel.</p> <p><strong>2.</strong> Click on <em>Connectors</em>.</p> <p><strong>3.</strong> Search for <em>incoming webhooks</em> and then click on <em>Add</em>; this action allows you to set webhooks on the channel.</p> <p><img src="../../../static/images/blog_posts/webhooks/1-teams-setup-1.png" alt="" /></p> <p><strong>4.</strong> To create the webhook, go back to the three dots (…) in the top right of the channel and click on <em>Connectors</em>.</p> <p><strong>5.</strong> You will see <em>Incoming Webhook</em> listed as a connector; click on <em>Configure</em>.</p> <p><img src="../../../static/images/blog_posts/webhooks/2-teams-setup-2.png" alt="" /></p> <p><strong>6.</strong> On the configuration screen, give the webhook a name and click <em>Create</em>.</p> <p><strong>7.</strong> You will be taken to a new screen that has a webhook URL. Copy that URL: this is the target URL for your webhook you’ll configure in Nautobot.</p> <p><img src="../../../static/images/blog_posts/webhooks/3-teams-setup-3.png" alt="" /></p> <h2 id="configuring-the-webhook-in-nautobot">Configuring the Webhook in Nautobot</h2> <p>Webhooks are an <em>Extensibility</em> feature in Nautobot, and configuration is quite simple. To create the webhook for this example:</p> <p><strong>1.</strong> Navigate to <em>Extensibility –&gt; Webhooks –&gt; +</em></p> <p><strong>2.</strong> Fill out the form, including:</p> <ul> <li>Name</li> <li>Specify the object(s) in the multi-selection drop-down menu (in our example the only object is <strong>DCIM | Device</strong>)</li> <li>Enable the webhook</li> <li>Specify the criteria for sending the webhook (<code class="language-plaintext highlighter-rouge">create</code>/<code class="language-plaintext highlighter-rouge">update</code>/<code class="language-plaintext highlighter-rouge">delete</code>) - select <code class="language-plaintext highlighter-rouge">create</code> for our example</li> <li>Populate the <em>URL</em> section with the target URL that MS Teams gave us</li> <li>Populate the <em>Body template</em> section with the JSON <code class="language-plaintext highlighter-rouge">{ "text":"data" }</code> for now (a later segment of this article will cover this section in more detail)</li> <li>Click <code class="language-plaintext highlighter-rouge">Create</code></li> </ul> <p><img src="../../../static/images/blog_posts/webhooks/4.2.png" alt="" /></p> <p>At this point, we have a very simple webhook:</p> <p><img src="../../../static/images/blog_posts/webhooks/5.1.png" alt="" /></p> <p>Let’s test it! Create a new device in Nautobot by navigating to <em>Devices –&gt; Devices –&gt; +</em> in the top-level Nautobot menu. Fill out the <em>Add a new device</em> form and click on the <code class="language-plaintext highlighter-rouge">Create</code> button. Your MS Teams channel should show a message that looks similar to this:</p> <p><img src="../../../static/images/blog_posts/webhooks/5.2.png" alt="" /></p> <blockquote> <p>The <a href="#appendix-full-device-data">Full Device Data Appendix</a> has the complete JSON returned by the <em>data</em> context environment variable.</p> </blockquote> <p>For the <em>Body template</em> section, we just included the <code class="language-plaintext highlighter-rouge">data</code> context variable for now, since that contains the richest set of information. The Nautobot documentation has more detailed information on <a href="https://nautobot.readthedocs.io/en/latest/models/extras/webhook/#configuration">webhook configuration</a> and <a href="https://nautobot.readthedocs.io/en/latest/models/extras/webhook/#jinja2-template-support">Jinja2 template support</a>.</p> <p>Since we sent all the <code class="language-plaintext highlighter-rouge">data</code> context variable to MS Teams, the output in our channel is a dump of the JSON within <code class="language-plaintext highlighter-rouge">data</code>. This is handy output to start because it gives us a visual of what attributes are present and what we can parse for. The next section digs into this and the notion of <em>context variables</em> a bit more.</p> <h3 id="the-body-template-section">The <em>Body template</em> Section</h3> <p>Let’s dig a bit deeper into the <em>Body template</em> section. This part of the webhook carries information to the receiving endpoint URL. Nautobot makes several context variables available for use in this section. The available context variables include: <code class="language-plaintext highlighter-rouge">event</code>, <code class="language-plaintext highlighter-rouge">model</code>, <code class="language-plaintext highlighter-rouge">timestamp</code>, <code class="language-plaintext highlighter-rouge">username</code>, <code class="language-plaintext highlighter-rouge">request_id</code>, and <code class="language-plaintext highlighter-rouge">data</code>.</p> <p>The entire <em>Body template</em> section (as well as the <em>Additional headers</em> section) supports <em>Jinja2</em> templating. The user can access and parse the context variables via Jinja2 formatting.</p> <p>If this section is left blank, Nautobot will populate the request body with a raw dump of the webhook context. Many platforms will not accept a pure JSON dump, as they require specific formatting. For example, MS Teams and Slack require the webhook info to be in a key, value pair format, with <code class="language-plaintext highlighter-rouge">"text"</code> as the key.</p> <p>So, as an example, to send MS Teams the device name, and the username that created the device, the body template would be:</p> <p><code class="language-plaintext highlighter-rouge">{ "text":"{{ data.name }} created by {{ username }}"}</code></p> <p>Using info within the <code class="language-plaintext highlighter-rouge">data</code> and <code class="language-plaintext highlighter-rouge">username</code> context variables, let’s craft a more descriptive message to send to MS Teams:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"text"</span><span class="p">:</span><span class="s2">"New device created in Nautobot. Device '{{ data.name }}' created in site '{{ data.site.name }}' with status '{{ data.status.value }}' on '{{ data.created }}' by user '{{ username }}'"</span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Update your webhook <em>body template</em> section with this data and create another device in Nautobot; you will see a more informative message in MS Teams:</p> <p><img src="../../../static/images/blog_posts/webhooks/6-msteams-msg.png" alt="" /></p> <h2 id="webhooks-and-chatops-symmetry">Webhooks and ChatOps Symmetry</h2> <p>For those familiar with <a href="https://github.com/nautobot/nautobot-plugin-chatops">Nautobot’s ChatOps</a> app and for those who may not be yet, I’d like to point out a symmetry here:</p> <ul> <li>ChatOps allows a user to query Nautobot for information and to initiate other actions within Nautobot from a chat platform (Slack, MS Teams, Webex Teams, Mattermost).</li> <li>Nautobot’s webhooks allow Nautobot to proactively communicate to users in their chat platform(s) when specific events happen.</li> </ul> <p><img src="../../../static/images/blog_posts/webhooks/7-symmetry.png" alt="" /></p> <p>These two actions together form a back-and-forth synergy, making interaction with Nautobot more efficient.</p> <h2 id="wrapping-up">Wrapping Up</h2> <p>Webhooks can play an important role in workflows because they coordinate activities between applications. As a central part of your automation infrastructure, Nautobot’s webhooks feature gives you another option for integration that best suits your environment.</p> <p>Thank you for your time, and have a Happy New Year!</p> <p>-Tim Fiola</p> <p>Developer Advocate</p> <h2 id="appendix-full-device-data">Appendix: Full Device Data</h2> <p>This is the full <code class="language-plaintext highlighter-rouge">json</code> output for the newly created device:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="err">'id':</span><span class="w"> </span><span class="err">'</span><span class="mi">8</span><span class="err">d</span><span class="mi">40</span><span class="err">f</span><span class="mi">3</span><span class="err">b</span><span class="mi">3-5</span><span class="err">a</span><span class="mi">04-413</span><span class="err">f-bee</span><span class="mi">0</span><span class="err">-c</span><span class="mi">6</span><span class="err">ff</span><span class="mi">312</span><span class="err">bbed</span><span class="mi">1</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="err">'url':</span><span class="w"> </span><span class="err">'/api/dcim/devices/</span><span class="mi">8</span><span class="err">d</span><span class="mi">40</span><span class="err">f</span><span class="mi">3</span><span class="err">b</span><span class="mi">3-5</span><span class="err">a</span><span class="mi">04-413</span><span class="err">f-bee</span><span class="mi">0</span><span class="err">-c</span><span class="mi">6</span><span class="err">ff</span><span class="mi">312</span><span class="err">bbed</span><span class="mi">1</span><span class="err">/'</span><span class="p">,</span><span class="w"> </span><span class="err">'name':</span><span class="w"> </span><span class="err">'web-test</span><span class="mi">-01</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="err">'device_type':</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">'id':</span><span class="w"> </span><span class="err">'</span><span class="mi">63</span><span class="err">d</span><span class="mi">6</span><span class="err">b</span><span class="mi">13</span><span class="err">d-e</span><span class="mi">2</span><span class="err">ab</span><span class="mi">-42</span><span class="err">b</span><span class="mi">9</span><span class="err">-a</span><span class="mi">847</span><span class="err">-eb</span><span class="mi">218708</span><span class="err">dc</span><span class="mi">3</span><span class="err">a'</span><span class="p">,</span><span class="w"> </span><span class="err">'url':</span><span class="w"> </span><span class="err">'/api/dcim/device-types/</span><span class="mi">63</span><span class="err">d</span><span class="mi">6</span><span class="err">b</span><span class="mi">13</span><span class="err">d-e</span><span class="mi">2</span><span class="err">ab</span><span class="mi">-42</span><span class="err">b</span><span class="mi">9</span><span class="err">-a</span><span class="mi">847</span><span class="err">-eb</span><span class="mi">218708</span><span class="err">dc</span><span class="mi">3</span><span class="err">a/'</span><span class="p">,</span><span class="w"> </span><span class="err">'manufacturer':</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">'id':</span><span class="w"> </span><span class="err">'</span><span class="mi">9843</span><span class="err">c</span><span class="mi">7</span><span class="err">a</span><span class="mi">4-6139-480</span><span class="err">f</span><span class="mi">-879</span><span class="err">c-e</span><span class="mi">012</span><span class="err">bcb</span><span class="mi">5</span><span class="err">ae</span><span class="mi">34</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="err">'url':</span><span class="w"> </span><span class="err">'/api/dcim/manufacturers/</span><span class="mi">9843</span><span class="err">c</span><span class="mi">7</span><span class="err">a</span><span class="mi">4-6139-480</span><span class="err">f</span><span class="mi">-879</span><span class="err">c-e</span><span class="mi">012</span><span class="err">bcb</span><span class="mi">5</span><span class="err">ae</span><span class="mi">34</span><span class="err">/'</span><span class="p">,</span><span class="w"> </span><span class="err">'name':</span><span class="w"> </span><span class="err">'Cisco'</span><span class="p">,</span><span class="w"> </span><span class="err">'slug':</span><span class="w"> </span><span class="err">'cisco'</span><span class="p">,</span><span class="w"> </span><span class="err">'display':</span><span class="w"> </span><span class="err">'Cisco'</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">'model':</span><span class="w"> </span><span class="err">'Nexus</span><span class="w"> </span><span class="mi">9</span><span class="err">Kv'</span><span class="p">,</span><span class="w"> </span><span class="err">'slug':</span><span class="w"> </span><span class="err">'cisco-nx-osv-chassis'</span><span class="p">,</span><span class="w"> </span><span class="err">'display':</span><span class="w"> </span><span class="err">'Cisco</span><span class="w"> </span><span class="err">Nexus</span><span class="w"> </span><span class="mi">9</span><span class="err">Kv'</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">'device_role':</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">'id':</span><span class="w"> </span><span class="err">'bc</span><span class="mi">1</span><span class="err">fffae-e</span><span class="mi">7e5-426</span><span class="err">c-a</span><span class="mi">08</span><span class="err">f-b</span><span class="mi">9</span><span class="err">b</span><span class="mi">5</span><span class="err">a</span><span class="mi">986</span><span class="err">bab</span><span class="mi">3</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="err">'url':</span><span class="w"> </span><span class="err">'/api/dcim/device-roles/bc</span><span class="mi">1</span><span class="err">fffae-e</span><span class="mi">7e5-426</span><span class="err">c-a</span><span class="mi">08</span><span class="err">f-b</span><span class="mi">9</span><span class="err">b</span><span class="mi">5</span><span class="err">a</span><span class="mi">986</span><span class="err">bab</span><span class="mi">3</span><span class="err">/'</span><span class="p">,</span><span class="w"> </span><span class="err">'name':</span><span class="w"> </span><span class="err">'Backbone'</span><span class="p">,</span><span class="w"> </span><span class="err">'slug':</span><span class="w"> </span><span class="err">'backbone'</span><span class="p">,</span><span class="w"> </span><span class="err">'display':</span><span class="w"> </span><span class="err">'Backbone'</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">'tenant':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'platform':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'serial':</span><span class="w"> </span><span class="err">''</span><span class="p">,</span><span class="w"> </span><span class="err">'asset_tag':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'site':</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">'id':</span><span class="w"> </span><span class="err">'</span><span class="mi">9117</span><span class="err">f</span><span class="mi">79</span><span class="err">b</span><span class="mi">-148</span><span class="err">b</span><span class="mi">-47</span><span class="err">f</span><span class="mi">6-9</span><span class="err">d</span><span class="mi">71</span><span class="err">-e</span><span class="mi">984</span><span class="err">d</span><span class="mi">602</span><span class="err">f</span><span class="mi">1</span><span class="err">ed'</span><span class="p">,</span><span class="w"> </span><span class="err">'url':</span><span class="w"> </span><span class="err">'/api/dcim/sites/</span><span class="mi">9117</span><span class="err">f</span><span class="mi">79</span><span class="err">b</span><span class="mi">-148</span><span class="err">b</span><span class="mi">-47</span><span class="err">f</span><span class="mi">6-9</span><span class="err">d</span><span class="mi">71</span><span class="err">-e</span><span class="mi">984</span><span class="err">d</span><span class="mi">602</span><span class="err">f</span><span class="mi">1</span><span class="err">ed/'</span><span class="p">,</span><span class="w"> </span><span class="err">'name':</span><span class="w"> </span><span class="err">'Jersey</span><span class="w"> </span><span class="err">City'</span><span class="p">,</span><span class="w"> </span><span class="err">'slug':</span><span class="w"> </span><span class="err">'jcy'</span><span class="p">,</span><span class="w"> </span><span class="err">'display':</span><span class="w"> </span><span class="err">'Jersey</span><span class="w"> </span><span class="err">City'</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">'rack':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'position':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'face':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'parent_device':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'status':</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">'value':</span><span class="w"> </span><span class="err">'active'</span><span class="p">,</span><span class="w"> </span><span class="err">'label':</span><span class="w"> </span><span class="err">'Active'</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">'primary_ip':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'primary_ip</span><span class="mi">4</span><span class="err">':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'primary_ip</span><span class="mi">6</span><span class="err">':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'secrets_group':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'cluster':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'virtual_chassis':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'vc_position':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'vc_priority':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'comments':</span><span class="w"> </span><span class="err">''</span><span class="p">,</span><span class="w"> </span><span class="err">'local_context_schema':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'local_context_data':</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="err">'tags':</span><span class="w"> </span><span class="p">[],</span><span class="w"> </span><span class="err">'custom_fields':</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="err">'created':</span><span class="w"> </span><span class="err">'</span><span class="mi">2021-12-14</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="err">'last_updated':</span><span class="w"> </span><span class="err">'</span><span class="mi">2021-12-14</span><span class="err">T</span><span class="mi">17</span><span class="err">:</span><span class="mi">31</span><span class="err">:</span><span class="mf">23.589832</span><span class="err">Z'</span><span class="p">,</span><span class="w"> </span><span class="err">'display':</span><span class="w"> </span><span class="err">'web-test</span><span class="mi">-01</span><span class="err">'</span><span class="p">}</span><span class="w"> </span></code></pre></div></div>Tim FiolaThis week’s Nautobot blog deals with webhooks. In basic terms, a webhook is a method for one web application to programmatically provide information to another web app.Building a Nautobot SSoT App2021-12-22T00:00:00+00:002021-12-22T00:00:00+00:00https://blog.networktocode.com/post/building-a-nautobot-ssot-app<p>In a <a href="https://blog.networktocode.com/post/nautobot-ssot-plugin/">previous post</a> we established the importance of having a <em>single source of truth</em> (SSoT), provided an overview of the Nautobot Single Source of Truth (SSoT) framework, and how the SSoT framework works to enable synchronization of your Systems of Record (SoR). In addition, we’ve also shown a Nautobot SSoT App for <a href="https://blog.networktocode.com/post/nautobot-plugin-ssot-arista/">Arista CloudVision</a> which extends the SSoT base framework. So now you ask, how do I synchronize my data to and from Nautobot using the Single Source of Truth framework? In this first part of a two-part series, I’ll be explaining the basics of creating your own SSoT app, and then next month, I’ll be following up with more advanced options available when building an SSoT app.</p> <blockquote> <p>Please note: it is expected that you’ve read the <code class="language-plaintext highlighter-rouge">Nautobot Plugin: Single Source of Truth (SSoT)</code> post and understand the framework terminology, such as Data Source and Data Target.</p> </blockquote> <p>The first thing to do when creating an SSoT app is to define the data shared between your SoR that you want to synchronize. For example, you might want to pull your Rooms from Device42 into Nautobot. You would then create a class that inherits from the DiffSyncModel class as shown below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSyncModel</span> <span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span> <span class="k">class</span> <span class="nc">Room</span><span class="p">(</span><span class="n">DiffSyncModel</span><span class="p">):</span> <span class="s">"""Room model."""</span> <span class="n">_modelname</span> <span class="o">=</span> <span class="s">"room"</span> <span class="n">_identifiers</span> <span class="o">=</span> <span class="p">(</span><span class="s">"name"</span><span class="p">,</span> <span class="s">"building"</span><span class="p">)</span> <span class="n">_shortname</span> <span class="o">=</span> <span class="p">(</span><span class="s">"name"</span><span class="p">,)</span> <span class="n">_attributes</span> <span class="o">=</span> <span class="p">(</span><span class="s">"notes"</span><span class="p">,)</span> <span class="n">_children</span> <span class="o">=</span> <span class="p">{</span><span class="s">"rack"</span><span class="p">:</span> <span class="s">"racks"</span><span class="p">}</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="n">building</span><span class="p">:</span> <span class="nb">str</span> <span class="n">notes</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="n">racks</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="s">"Rack"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">()</span> </code></pre></div></div> <p>You’ll notice that there are both public and private class attributes defined. Each private attribute is used to help define the model itself within the DiffSync framework. There are two attributes required on every DiffSyncModel, the <code class="language-plaintext highlighter-rouge">_modelname</code> and <code class="language-plaintext highlighter-rouge">_identifiers</code> attributes. The <code class="language-plaintext highlighter-rouge">_modelname</code> attribute defines the type of the model and is used to identify the shared models between your SoR. The <code class="language-plaintext highlighter-rouge">_identifier</code> attribute specifies the public attributes used to generate a name for the objects created when loading data from your adapters. It’s essential to confirm that the identifiers used for the object make it globally unique to ensure an accurate sync.</p> <p>The remaining attributes are optional but can be quite useful in the process. The <code class="language-plaintext highlighter-rouge">_shortname</code> attribute identifies an object apart from other objects of the same type allowing for use of a shorter name. The <code class="language-plaintext highlighter-rouge">_attributes</code> attribute specifies all attributes that are of interest for synchronization. You’ll notice that each of these public attributes is defined using <a href="https://pydantic-docs.helpmanual.io/">pydantic</a> typing syntax. These are essential for ensuring data integrity while performing the synchronization. Please note that you must use the <code class="language-plaintext highlighter-rouge">Optional</code> type for any attribute that you wish to allow to be <code class="language-plaintext highlighter-rouge">None</code>. The last private attribute is <code class="language-plaintext highlighter-rouge">_children</code>, which defines other models related to the model you’re creating. In this example, Rooms are children of the Building as you have many Rooms inside a Building. This allows you to define a hierarchy of models for importing. Please note that this is meant for a direct parent-to-child relationship and not multi-branching inheritance. The <code class="language-plaintext highlighter-rouge">_children</code> attribute is defined using the pattern of <code class="language-plaintext highlighter-rouge">{&lt;model_name&gt;: &lt;field_name&gt;}</code>.</p> <p>The next step is to define the CRUD (Create, Update, Delete) methods for each model. These methods will handle taking the data, once loaded from your Data Source, and making the relevant changes to the object in your Data Target. Although you may add the CRUD methods for your object to the DiffSyncModel class that you created in the first step, best practice is to create new classes that inherit from that DiffSyncModel class, as shown below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.utils.text</span> <span class="kn">import</span> <span class="n">slugify</span> <span class="kn">from</span> <span class="nn">nautobot.dcim.models</span> <span class="kn">import</span> <span class="n">RackGroup</span> <span class="k">as</span> <span class="n">NautobotRackGroup</span> <span class="kn">from</span> <span class="nn">nautobot.dcim.models</span> <span class="kn">import</span> <span class="n">Site</span> <span class="k">as</span> <span class="n">NautobotSite</span> <span class="k">class</span> <span class="nc">NautobotRoom</span><span class="p">(</span><span class="n">Room</span><span class="p">):</span> <span class="s">"""Nautobot Room CRUD methods."""</span> <span class="o">@</span><span class="nb">classmethod</span> <span class="k">def</span> <span class="nf">create</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">diffsync</span><span class="p">,</span> <span class="n">ids</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="s">"""Create RackGroup object in Nautobot."""</span> <span class="n">new_rg</span> <span class="o">=</span> <span class="n">NautobotRackGroup</span><span class="p">(</span> <span class="n">name</span><span class="o">=</span><span class="n">ids</span><span class="p">[</span><span class="s">"name"</span><span class="p">],</span> <span class="n">slug</span><span class="o">=</span><span class="n">slugify</span><span class="p">(</span><span class="n">ids</span><span class="p">[</span><span class="s">"name"</span><span class="p">]),</span> <span class="n">site</span><span class="o">=</span><span class="n">NautobotSite</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">ids</span><span class="p">[</span><span class="s">"building"</span><span class="p">]),</span> <span class="n">description</span><span class="o">=</span><span class="n">attrs</span><span class="p">[</span><span class="s">"notes"</span><span class="p">]</span> <span class="k">if</span> <span class="n">attrs</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"notes"</span><span class="p">)</span> <span class="k">else</span> <span class="s">""</span><span class="p">,</span> <span class="p">)</span> <span class="n">new_rg</span><span class="p">.</span><span class="n">validated_save</span><span class="p">()</span> <span class="k">return</span> <span class="nb">super</span><span class="p">().</span><span class="n">create</span><span class="p">(</span><span class="n">ids</span><span class="o">=</span><span class="n">ids</span><span class="p">,</span> <span class="n">diffsync</span><span class="o">=</span><span class="n">diffsync</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="n">attrs</span><span class="p">)</span> <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="s">"""Update RackGroup object in Nautobot."""</span> <span class="n">_rg</span> <span class="o">=</span> <span class="n">NautobotRackGroup</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">site__name</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">building</span><span class="p">)</span> <span class="k">if</span> <span class="n">attrs</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"notes"</span><span class="p">):</span> <span class="n">_rg</span><span class="p">.</span><span class="n">description</span> <span class="o">=</span> <span class="n">attrs</span><span class="p">[</span><span class="s">"notes"</span><span class="p">]</span> <span class="n">_rg</span><span class="p">.</span><span class="n">validated_save</span><span class="p">()</span> <span class="k">return</span> <span class="nb">super</span><span class="p">().</span><span class="n">update</span><span class="p">(</span><span class="n">attrs</span><span class="p">)</span> <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="s">"""Delete RackGroup object from Nautobot."""</span> <span class="bp">self</span><span class="p">.</span><span class="n">diffsync</span><span class="p">.</span><span class="n">job</span><span class="p">.</span><span class="n">log_warning</span><span class="p">(</span><span class="sa">f</span><span class="s">"RackGroup </span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s"> will be deleted."</span><span class="p">)</span> <span class="nb">super</span><span class="p">().</span><span class="n">delete</span><span class="p">()</span> <span class="n">rackgroup</span> <span class="o">=</span> <span class="n">NautobotRackGroup</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="o">**</span><span class="bp">self</span><span class="p">.</span><span class="n">get_identifiers</span><span class="p">())</span> <span class="n">rackgroup</span><span class="p">.</span><span class="n">delete</span><span class="p">()</span> <span class="k">return</span> <span class="bp">self</span> </code></pre></div></div> <p>Each of the <code class="language-plaintext highlighter-rouge">create()</code>, <code class="language-plaintext highlighter-rouge">update()</code>, and <code class="language-plaintext highlighter-rouge">delete()</code> methods for an object are called once a diff is completed and the synchronization process is started. Which method is called depends upon the required changes to the object in your Data Target. When the <code class="language-plaintext highlighter-rouge">create()</code> method is called, the object’s identifier and other attributes are passed to it as the <code class="language-plaintext highlighter-rouge">ids</code> and <code class="language-plaintext highlighter-rouge">attrs</code> variables respectively. The diffsync variable is for handling interactions with the DiffSync Job, such as sending log messages. For the logging of the Job results to be accurate, it is essential that the object is returned to the create method with the variables passed. However, unlike the <code class="language-plaintext highlighter-rouge">create()</code> method, the <code class="language-plaintext highlighter-rouge">update()</code> method receives only the attributes that have been changed for an object. This means that it is required for the implementer to validate if attributes have been passed or not before making appropriate changes. The <code class="language-plaintext highlighter-rouge">delete()</code> method will receive only the class object itself.</p> <blockquote> <p>When utilizing inheritance between models, ensure the related models have the <code class="language-plaintext highlighter-rouge">update_forward_refs()</code> method called. This is essential to establish the relationships between objects.</p> </blockquote> <p>Once the models and their CRUD methods have been defined, the next step is to write the adapters that load the models you specified in the previous steps. It is in this sense that the adapter class <em>adapts</em> the data from your Data Source. The adapters are required to reference each model that you wish to have considered at the top of the DiffSync object along with a <code class="language-plaintext highlighter-rouge">top_level</code> list of your models in the order that you wish to have them processed, as you can see below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">Building</span><span class="p">,</span> <span class="n">Room</span> <span class="k">class</span> <span class="nc">Device42Adapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="s">"""DiffSync adapter for Device42 server."""</span> <span class="n">building</span> <span class="o">=</span> <span class="n">Building</span> <span class="n">room</span> <span class="o">=</span> <span class="n">Room</span> <span class="n">top_level</span> <span class="o">=</span> <span class="p">[</span><span class="s">"building"</span><span class="p">]</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span> <span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">Building</span><span class="p">,</span> <span class="n">Room</span> <span class="k">class</span> <span class="nc">NautobotAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="s">"""Nautobot adapter for DiffSync."""</span> <span class="n">building</span> <span class="o">=</span> <span class="n">Building</span> <span class="n">room</span> <span class="o">=</span> <span class="n">Room</span> <span class="n">top_level</span> <span class="o">=</span> <span class="p">[</span><span class="s">"building"</span><span class="p">]</span> </code></pre></div></div> <p>As you can see above, you will always have two Systems of Record in a diff so you will need an adapter for both. It is best practice to have them matching at the top to ensure that items are processed identically. As you can see in the examples above, only the Building model is in the <code class="language-plaintext highlighter-rouge">top_level</code> list as the Room model is a child and will be processed after the Building is. It is up to the implementer to determine how they wish to load the models they create in the adapters. While loading your models from the methods in your adapters, it is essential that you pass valid DiffSyncModel objects that adhere to what you specified in your models when passed to the <code class="language-plaintext highlighter-rouge">add()</code> function. Failing to do so will cause validation errors.</p> <blockquote> <p>It’s advised to use a <code class="language-plaintext highlighter-rouge">load()</code> method to call your other model-specific methods to keep things concise.</p> </blockquote> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync.exceptions</span> <span class="kn">import</span> <span class="n">ObjectAlreadyExists</span> <span class="k">class</span> <span class="nc">Device42Adapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="p">...</span> <span class="k">def</span> <span class="nf">load_rooms</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="s">"""Load Device42 rooms."""</span> <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">_device42</span><span class="p">.</span><span class="n">api_call</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="s">"api/1.0/rooms"</span><span class="p">)[</span><span class="s">"rooms"</span><span class="p">]:</span> <span class="k">if</span> <span class="n">record</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"building"</span><span class="p">):</span> <span class="n">room</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">room</span><span class="p">(</span> <span class="n">name</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s">"name"</span><span class="p">],</span> <span class="n">building</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s">"building"</span><span class="p">],</span> <span class="n">notes</span><span class="o">=</span><span class="n">record</span><span class="p">[</span><span class="s">"notes"</span><span class="p">]</span> <span class="k">if</span> <span class="n">record</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"notes"</span><span class="p">)</span> <span class="k">else</span> <span class="s">""</span><span class="p">,</span> <span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">room</span><span class="p">)</span> <span class="n">_site</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">building</span><span class="p">,</span> <span class="n">record</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"building"</span><span class="p">))</span> <span class="n">_site</span><span class="p">.</span><span class="n">add_child</span><span class="p">(</span><span class="n">child</span><span class="o">=</span><span class="n">room</span><span class="p">)</span> <span class="k">except</span> <span class="n">ObjectAlreadyExists</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">job</span><span class="p">.</span><span class="n">log_warning</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">record</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span><span class="si">}</span><span class="s"> is already loaded. </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">job</span><span class="p">.</span><span class="n">log_warning</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">record</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span><span class="si">}</span><span class="s"> missing Building, skipping."</span><span class="p">)</span> <span class="k">continue</span> </code></pre></div></div> <p>The example above shows how data is pulled from the Device42 API and creates the Room objects that were detailed in the first step. Once the object has been created, it is then <em>added</em> into the DiffSync set with the <code class="language-plaintext highlighter-rouge">add()</code> method. As a Room is a child object of a Building, there is an additional step of finding the parent Building object with the <code class="language-plaintext highlighter-rouge">get()</code> method, and then using the <code class="language-plaintext highlighter-rouge">add_child()</code> method to add the relationship between the objects. If there is an existing object with the same identifiers, the <code class="language-plaintext highlighter-rouge">ObjectAlreadyExists</code> exception will be thrown, so it’s advised to wrap the <code class="language-plaintext highlighter-rouge">add()</code> method in a try/except block.</p> <p>With your adapters for each SoR created, the final step is to write your <a href="https://nautobot.readthedocs.io/en/latest/additional-features/jobs/">Nautobot Job</a>. This will handle the loading of your models from the adapters, the diff of the objects once loaded, and the synchronization of data by calling the CRUD methods as appropriate. The Job class must be derived from either the <code class="language-plaintext highlighter-rouge">DataSource</code> or <code class="language-plaintext highlighter-rouge">DataTarget</code> class and is required to include a <code class="language-plaintext highlighter-rouge">sync_data</code> method to handle the synchronization process. Optionally, you can also add a <code class="language-plaintext highlighter-rouge">config_information</code> or <code class="language-plaintext highlighter-rouge">data_mappings</code> method to enrich the data presented to the end user in Nautobot.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">django.templatetags.static</span> <span class="kn">import</span> <span class="n">static</span> <span class="kn">from</span> <span class="nn">nautobot.extras.jobs</span> <span class="kn">import</span> <span class="n">Job</span> <span class="kn">from</span> <span class="nn">nautobot_ssot.jobs.base</span> <span class="kn">import</span> <span class="n">DataSource</span> <span class="kn">from</span> <span class="nn">diffsync.exceptions</span> <span class="kn">import</span> <span class="n">ObjectNotCreated</span> <span class="kn">from</span> <span class="nn">.device42</span> <span class="kn">import</span> <span class="n">Device42Adapter</span> <span class="kn">from</span> <span class="nn">.nautobot</span> <span class="kn">import</span> <span class="n">NautobotAdapter</span> <span class="k">class</span> <span class="nc">Device42DataSource</span><span class="p">(</span><span class="n">DataSource</span><span class="p">,</span> <span class="n">Job</span><span class="p">):</span> <span class="s">"""Device42 SSoT Data Source."""</span> <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span> <span class="s">"""Meta data for Device42."""</span> <span class="n">name</span> <span class="o">=</span> <span class="s">"Device42"</span> <span class="n">data_source</span> <span class="o">=</span> <span class="s">"Device42"</span> <span class="n">data_source_icon</span> <span class="o">=</span> <span class="n">static</span><span class="p">(</span><span class="s">"./d42_logo.png"</span><span class="p">)</span> <span class="n">description</span> <span class="o">=</span> <span class="s">"Sync information from Device42 to Nautobot"</span> <span class="k">def</span> <span class="nf">sync_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="s">"""Device42 Sync."""</span> <span class="n">d42_adapter</span> <span class="o">=</span> <span class="n">Device42Adapter</span><span class="p">(</span><span class="n">job</span><span class="o">=</span><span class="bp">self</span><span class="p">,</span> <span class="n">sync</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">)</span> <span class="n">d42_adapter</span><span class="p">.</span><span class="n">load</span><span class="p">()</span> <span class="n">nb_adapter</span> <span class="o">=</span> <span class="n">NautobotAdapter</span><span class="p">(</span><span class="n">job</span><span class="o">=</span><span class="bp">self</span><span class="p">,</span> <span class="n">sync</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">)</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">load</span><span class="p">()</span> <span class="n">diff</span> <span class="o">=</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">diff_from</span><span class="p">(</span><span class="n">d42_adapter</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">.</span><span class="n">diff</span> <span class="o">=</span> <span class="n">diff</span><span class="p">.</span><span class="nb">dict</span><span class="p">()</span> <span class="bp">self</span><span class="p">.</span><span class="n">sync</span><span class="p">.</span><span class="n">save</span><span class="p">()</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_info</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="n">diff</span><span class="p">.</span><span class="n">summary</span><span class="p">())</span> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="p">.</span><span class="n">kwargs</span><span class="p">[</span><span class="s">"dry_run"</span><span class="p">]:</span> <span class="k">try</span><span class="p">:</span> <span class="n">nb_adapter</span><span class="p">.</span><span class="n">sync_from</span><span class="p">(</span><span class="n">d42_adapter</span><span class="p">)</span> <span class="k">except</span> <span class="n">ObjectNotCreated</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_debug</span><span class="p">(</span><span class="sa">f</span><span class="s">"Unable to create object. </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_success</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="s">"Sync complete."</span><span class="p">)</span> <span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span><span class="n">Device42DataSource</span><span class="p">]</span> </code></pre></div></div> <p>As is shown, the Job should create an instance of each of your adapters and call their <code class="language-plaintext highlighter-rouge">load()</code> methods to create the DiffSyncModel objects. Once that’s done, a diff and sync can be completed utilizing either the <code class="language-plaintext highlighter-rouge">diff_from</code>/<code class="language-plaintext highlighter-rouge">diff_to</code> and <code class="language-plaintext highlighter-rouge">sync_from</code>/<code class="language-plaintext highlighter-rouge">sync_to</code> methods on the adapter objects. Which you use depends upon which way you wish to have the synchronization performed. In the example above, once the models have been loaded, a diff from Device42 to Nautobot is done to report the required objects to be created, updated, or deleted. Finally, if the Job is not a dry-run, synchronization will be executed. Again, this is done with a <code class="language-plaintext highlighter-rouge">sync_from</code> from the Device42 adapter object to the Nautobot adapter object. Depending upon the results of an object being created, an <code class="language-plaintext highlighter-rouge">ObjectNotCreated</code> exception may be thrown, so it’s advised to use a try/except block when calling the sync methods to ensure it’s caught and handled appropriately. Once all of the objects have been processed, a final success log is sent and the GUI should be updated to reflect the changes.</p> <p>In summary, the creation of an SSoT app requires the following steps to be completed:</p> <ol> <li> <p>Create one or more DiffSyncModel classes to define the data you wish to synchronize.</p> </li> <li> <p>For each DiffSyncModel, define the CRUD methods to handle the requisite changes in your Data Target.</p> </li> <li> <p>Write a DiffSync adapter class for each System of Record to load data from each into your DiffSyncModel classes.</p> </li> <li> <p>Finally, write your Nautobot Job to perform the synchronization of data between your Data Source and Data Target.</p> </li> </ol> <p>In the next part of this series, we’ll look into how to customize the processing of objects and the use of global and model flags in the DiffSync process.</p> <p>-Justin</p>Justin DrewIn a previous post we established the importance of having a single source of truth (SSoT), provided an overview of the Nautobot Single Source of Truth (SSoT) framework, and how the SSoT framework works to enable synchronization of your Systems of Record (SoR). In addition, we’ve also shown a Nautobot SSoT App for Arista CloudVision which extends the SSoT base framework. So now you ask, how do I synchronize my data to and from Nautobot using the Single Source of Truth framework? In this first part of a two-part series, I’ll be explaining the basics of creating your own SSoT app, and then next month, I’ll be following up with more advanced options available when building an SSoT app.Integrated Version Control in Nautobot2021-12-16T00:00:00+00:002021-12-16T00:00:00+00:00https://blog.networktocode.com/post/nautobot-version-control-roundup<p>In the past few months, many of us here at Network to Code have been investing a lot of time with <a href="https://github.com/nautobot/nautobot-plugin-version-control">Nautobot Version Control</a>. The ultimate goal with adding version control to Nautobot is to make network automation safer by bringing proven <em>Git</em>- and <em>GitHub</em>-like workflows directly to Nautobot.</p> <p>Along the way, we’ve created a lot of content in order to showcase how the version control capability can benefit Nautobot users and those looking for safer ways to manage data. This week’s post serves as an aggregation for all the Version Control related content to date.</p> <p>Below, organized by content type, are hyperlinks to each piece of content that has been produced over the past few months.</p> <h2 id="code-and-documentation">Code and Documentation</h2> <ul> <li><a href="https://github.com/nautobot/nautobot-plugin-version-control">Version Control repository</a> <ul> <li>This holds the open source code for the Version Control integration.</li> </ul> </li> <li><a href="https://nautobot-version-control.readthedocs.io/en/latest/">Version Control documentation on readthedocs.io</a> <ul> <li>These are the official docs for the integration.</li> </ul> </li> </ul> <h2 id="blogs">Blogs</h2> <ul> <li><a href="https://blog.networktocode.com/post/database-version-control-with-nautobot/">Database Version Control with Nautobot</a> <ul> <li>This blog discusses the value that Version Control brings with its Git-like workflows and how those workflows fit into existing Nautobot capabilities to keep network automation data clean.</li> </ul> </li> <li><a href="https://blog.networktocode.com/post/nautobots-rollback/">Nautobots, Roll Back!</a> <ul> <li>This post discusses the business and operational value in allowing users to review changes to SoT data prior to pushing them to production, and then to quickly undo that set of changes if needed.</li> </ul> </li> <li><a href="https://blog.networktocode.com/post/keeping-data-clean-in-nautobot/">Keeping Data Clean with Nautobot Config Contexts, Schemas, and Git</a> <ul> <li>This post is not Version Control related explicitly, but it does discuss Nautobot features you can leverage that complement the Version Control workflows.</li> <li>These features, especially when paired with Nautobot Version Control app workflows, provide a layered, <em>defense in depth</em> to keep your automation data clean.</li> </ul> </li> </ul> <h2 id="videos">Videos</h2> <ul> <li><a href="https://youtube.com/playlist?list=PLjA0bhxgryJ2oTSwPNX7djwAuL7NOYCVD">YouTube Version Control playlist</a> <ul> <li>Intro Series: This series of videos introduces the user to the Version Control with Nautobot, its use cases, and how to use the app. <ul> <li>Unit 1: Defining the problem at hand</li> <li>Unit 2: Bringing version control to Nautobot’s database</li> <li>Unit 3: Navigating the top-level Version Control menu</li> <li>Unit 4: Managing pull requests with Version Control</li> </ul> </li> <li>Demo Series: This series of videos provides demonstrations of how to set up a demo instance of Version Control with Nautobot, workflows enabled by it, and how to interact with Version Control programmatically. <ul> <li>Unit 1: Setting up a demo environment</li> <li>Unit 2: Version Control enabled workflow demo</li> <li>Unit 3: Version Control: Jenkins CI pipeline added</li> <li>Unit 4: Using the Version Control API</li> </ul> </li> </ul> </li> </ul> <p>In the coming months we hope to have more engagement to get feedback users and to further showcase the business and operational benefits a version-controlled Nautobot environment provides.</p> <p>Thank you! Have a great day, and have a <strong>great</strong> New Year!</p> <p>-Tim Fiola</p> <p>Developer Advocate</p>Tim FiolaIn the past few months, many of us here at Network to Code have been investing a lot of time with Nautobot Version Control. The ultimate goal with adding version control to Nautobot is to make network automation safer by bringing proven Git- and GitHub-like workflows directly to Nautobot.How a Job as an Automation Engineer Has Changed Decisions I Make in My Personal Life2021-12-14T00:00:00+00:002021-12-14T00:00:00+00:00https://blog.networktocode.com/post/automation-in-daily-life<p>Early on in my network journey I was tasked with data mining and bulk configuration change tasks, which were quite common tasks at the NOC I was working at that time. These weren’t tasks I was terribly pleased with doing after the third or fourth time doing the same exact thing over and over again. Things started to evolve from there, simple formulas in spreadsheets to transform data to a Java app that helped with incident management. With each solution I conjured up, my job became less repetitive and my perception on any repetitive tasks changed. A few chapters later in my career I moved on from one of solutions to building automation at scale.</p> <p>As any automation engineer will admit, the first step in automating a known workflow is thinking differently. This typically starts with evaluating <code class="language-plaintext highlighter-rouge">current</code> requirements; then reevaluating <code class="language-plaintext highlighter-rouge">legacy</code> requirements helps to shape the outcome. So many times have I spent weeks developing a solution only to find that requirement <code class="language-plaintext highlighter-rouge">X</code> exists because an engineer said or did something years ago, and no one has questioned whether this is still applicable. Once we’ve settled on what the new workflow will look like, the second part to thinking differently is focusing on trying to say yes, instead of how many reasons there may be to say no. A wireless engineer once told me it was impossible to have help desk troubleshoot a wireless connection via automation. After talking through the most common issues, I was able to create a playbook that was tied to a <code class="language-plaintext highlighter-rouge">Slack</code> chat command that covered the most common use case and prevented those tickets from making it to network ops.</p> <p>These simple working practices have now influenced decisions in my personal life. Now, when selecting almost any non-trivial purchase my first thought is <code class="language-plaintext highlighter-rouge">can I automate some portion or use technology to enrich my experience?</code> I have always had the mentality to solve with technology first, but typically that has led to shortsighted purpose-driven purchases.</p> <h2 id="aquarium">Aquarium</h2> <p>There is a running joke with my coworkers that I can’t write a <a href="../InfluxDB/">blog post</a> or do a presentation without making it about my aquarium at some point. To make sure I don’t disappoint, automation has been front-of-mind for so many purchases for my aquarium. This started with creating a controller based on a RaspberryPi and industrial sensors so that I could have better visibility into telemetry metrics, such as temperature and pH. After starting to collect data, I quickly found myself rethinking the equipment I had selected. This opened my eyes to the art of the possibility.</p> <p>About six months after starting my first aquarium build (in many years), I started to dream of what is next. How can I take what I learned to make more informed decisions that make my life easier and make a better ecosystem for my pets? I went from having to manually top off fresh water, cleaning filters every few days, and manually adding chemicals to a mostly hands-off approach to managing my aquarium.</p> <h3 id="things-improved-with-automation">Things Improved with Automation</h3> <ul> <li>Cleaning filter socks every few days -&gt; Automated roller mat that changes the filter media based on an optical sensor and only requires once-a-month maintenance.</li> <li>Daily water top offs of ~1 gallon of fresh water -&gt; Auto top off that uses a 10-gallon reservoir and optical sensor to ensure a more stable salinity.</li> <li>Telemetry statistics <ul> <li>Reagent-based tests taking roughly half an hour to complete in total, which led to never being done -&gt; Some done every 2 minutes while others are done every 6 hours, and all are stored in a time-series database with Grafana for visualization.</li> <li>Non-reagent test which takes seconds to perform BUT rarely did -&gt; Just like reagent-based, performed every 2 minutes and sent to the same time-series database.</li> </ul> </li> <li>Lighting was done with hard on/off timers which could be a harsh transition -&gt; Lighting based on a ramp sunrise/sunset/moonlight cycle to help promote happy pets and coral growth.</li> </ul> <h2 id="fireworks">Fireworks</h2> <p>Growing up in a small town, every year the Fourth of July was a big event for my family, and I was fascinated with setting off fireworks. As adults, my brother and I still have that childlike fascination. His yearly Fourth of July part started off with him manually setting up each firework one by one for the whole family to marvel at, but he wanted something more.</p> <p>Our first automated firework display was roughly 15 minutes long and was done the old-fashioned way. In the weeks leading up to the party my brother had spent countless hours watching YouTube videos of the specific fireworks he purchased and researching exact run times of each. When the time came to build the display, we had four sheets of plywood where he had mapped out each firework and the exact length of time delay fuse that was needed to trigger each one without overlap. It was a masterpiece and had a single point of failure…. One fuse path, nothing had multiple paths or a concept of failure domain. By some miracle, the whole show went from start to finish without issue, and only one mortar failed to launch.</p> <p>The next two years the display was driven by a remote firing system. This gave us the ability to stop mid-show, if needed (fire safety reasons were a concern), and we went from one failure domain to sixteen failure domains. Evolving the display was a big step forward to ensure success, plus it gave two adult “children” the ability to press buttons to blow things up, so it was a big win.</p> <h2 id="home-automation">Home Automation</h2> <p>This mindset of automation-first has also influenced purchases of light switches, garage door openers, TVs, audio systems, and even Christmas lights. Small tasks of having schedules for exterior lights or skills with Amazon Alexa have improved the ecosystem that is my home. With simply telling Alexa <code class="language-plaintext highlighter-rouge">It's bedtime</code> all of my interior lights &amp; TVs shutoff. This was a huge win living with someone who always forgets to turn things off.</p> <p>The choice for an audio system was something I pondered for quite some time. Finding a single room/purpose solution was easy but the right whole home audio experience that integrated with other ecosystems I had was the more difficult task. <code class="language-plaintext highlighter-rouge">AirPlay2</code> compatibility along with a built in multi-zone audio, multiple available form factors including indoor &amp; outdoor solutions, and Amazon Alexa integration were the tech-related requirements. In the end, the most comprehensive product that also had great audio quality ended up being on the higher end of pricing, but having the complete end-to-end integration was a huge deciding factor.</p> <p>My latest child-like obsession that has been influenced by automation has been my Christmas light display. This year is my second year of having a display driven via automation and choreographed to music. When I was a child there was one house that had a few props that would have sections that would blink to look like rotating tires on a truck or Santa’s waving at people as they passed by. This brought me so much joy and being able to build a display using a RaspberryPi, a soldering iron, and FM transmitter just to potentially bring others the same joy it brought me as a child is worth the expense.</p> <h2 id="summary">Summary</h2> <p>This post has been a departure from normal tech blogs, but I think it’s fascinating to see how what I do day in and day out for work has influenced my personal life. It’s very common to see people in tech use tech to make their lives easier, but I have found a higher concentration of an automation-first approach at home in automation engineers than others in tech. Always question why and how you do things—you may one day surprise yourself by realizing how automation could make things easier.</p> <p>-Jeremy</p>Jeremy WhiteEarly on in my network journey I was tasked with data mining and bulk configuration change tasks, which were quite common tasks at the NOC I was working at that time. These weren’t tasks I was terribly pleased with doing after the third or fourth time doing the same exact thing over and over again. Things started to evolve from there, simple formulas in spreadsheets to transform data to a Java app that helped with incident management. With each solution I conjured up, my job became less repetitive and my perception on any repetitive tasks changed. A few chapters later in my career I moved on from one of solutions to building automation at scale.Nautobot ChatOps for Grafana2021-12-09T00:00:00+00:002021-12-09T00:00:00+00:00https://blog.networktocode.com/post/nautobot-chatops-for-grafana<p>Two of the more intriguing topics I have heard lately that also seems to resonates with network engineers and network professionals is the insight telemetry provides, and the ease of use chat platforms such as Slack and Microsoft Teams deliver to your keyboard and fingertips. The Grafana ChatOps application is designed to provide the best of both worlds. Grafana ChatOps is a <a href="https://nautobot.readthedocs.io">Nautobot</a> extension used with the <a href="https://github.com/nautobot/nautobot-plugin-chatops/">Nautobot ChatOps</a> base framework to provide all the operational graphs provided by Grafana delivered via chat clients.</p> <p>Today, we will walk through some of the features within the Grafana ChatOps integration, as well as some of the requirements and procedures to get up and running with Grafana ChatOps.</p> <p>An important note on the architecture design choices with this ChatOps app (plugin) is that chat commands are defined dynamically based on the Grafana panels and dashboards (we’ll go into this a little later). When you launch the app for the first time, you will see that no chat commands have been defined yet. You can define commands automatically or manually and tie them to specific Grafana panels within a dashboard.</p> <h2 id="installation">Installation</h2> <p>The package for the Grafana ChatOps app is available on PyPI and can be installed using <a href="https://pypi.org/project/pip/">pip</a>.</p> <p>Prior to installing the Nautobot Grafana Plugin, you should have the following installed:</p> <ul> <li><a href="https://nautobot.readthedocs.io/en/stable/installation/nautobot/">Nautobot</a> installed and configured.</li> <li><a href="https://grafana.com/docs/grafana/latest/installation/">Grafana</a> application installed and configured with dashboards and panels.</li> <li><a href="https://grafana.com/docs/grafana/latest/administration/image_rendering/">Grafana Image Rendering Service</a> installed.</li> <li><a href="https://grafana.com/grafana/plugins/grafana-image-renderer/">Grafana Image Rendering Plugin for Grafana</a> installed in your Grafana application.</li> <li><a href="https://github.com/nautobot/nautobot-plugin-chatops/blob/develop/docs/chat_setup/chat_setup.md">Nautobot Plugin ChatOps</a> installed and configured for your specific chat platform.</li> </ul> <p>For the full installation guide, please refer to the <a href="https://github.com/nautobot/nautobot-plugin-chatops-grafana/blob/main/docs/installation.md">Grafana ChatOps repo Install Guide</a>.</p> <h2 id="usage">Usage</h2> <p>Building Grafana ChatOps commands can be done using a manual or automated approach. The automated approach uses the <a href="https://diffsync.readthedocs.io/en/latest/index.html">DiffSync</a> library to synchronize Grafana dashboards, panels, and variables with the Nautobot Grafana ChatOps plugin.</p> <h3 id="defining-commands">Defining Commands</h3> <p>To define a command within the Grafana plugin for use with your chat client, there are two main components that we need to have populated.</p> <ul> <li>Define at least one Grafana Dashboard.</li> <li>Define at least one Grafana Panel within the Dashboard.</li> </ul> <p>This tutorial will take you through the steps noted above to get a chat command exposed in your chat client.</p> <p>The first step is to define a dashboard so that the Grafana plugin is aware of the dashboard that exists within Grafana. You can define a dashboard in Grafana in two ways: defining a dashboard manually or using the “Sync” feature to synchronize your Grafana dashboards automatically.</p> <h4 id="defining-a-dashboard-manually">Defining a Dashboard Manually</h4> <p>To define a dashboard manually, you can go to <code class="language-plaintext highlighter-rouge">Plugins &gt; Dashboards</code> and click the <code class="language-plaintext highlighter-rouge">+ Add</code> button located in the upper right of the screen. In the form for a new dashboard, you need to define the <code class="language-plaintext highlighter-rouge">slug</code>, <code class="language-plaintext highlighter-rouge">uid</code>, and <code class="language-plaintext highlighter-rouge">Friendly Name</code>.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_grafana/dashboard_add.png" alt="New Dashboard" /></p> <blockquote> <p>NOTE: You can find the slug and uid info by navigating to your Grafana instance and going to the desired dashboard, <img src="../../../static/images/blog_posts/nautobot_chatops_grafana/dashboard_url.png" alt="New Dashboard" /></p> </blockquote> <h4 id="defining-a-dashboard-using-the-sync-method">Defining a Dashboard Using the Sync Method</h4> <p>Alternatively, you can define a set of dashboards by synchronizing your Grafana dashboard configuration to the Grafana plugin. To synchronize dashboards, within Nautobot, navigate to <code class="language-plaintext highlighter-rouge">Plugins &gt; Dashboards</code> and click the <code class="language-plaintext highlighter-rouge">Sync</code> button.</p> <p>This process will utilize the <a href="https://diffsync.readthedocs.io/en/latest/">DiffSync</a> library to synchronize, create, update, and delete dashboards in Nautobot with the Dashboards that are defined in the Grafana application. Once complete, you will see all dashboards imported into Nautobot.</p> <h3 id="defining-grafana-panels">Defining Grafana Panels</h3> <p>The second step to defining Grafana commands in Nautobot for your chat client is to define the panels you wish to expose via chat.</p> <blockquote> <p>Panels are closely associated to chat commands, where there will be a chat command for each panel defined.</p> </blockquote> <p>Similar to dashboards, you can define panels in two ways within Nautobot.</p> <h4 id="defining-a-panel-manually">Defining a Panel Manually</h4> <p>To define a panel manually, go to <code class="language-plaintext highlighter-rouge">Plugins &gt; Panels</code> and click the <code class="language-plaintext highlighter-rouge">+ Add</code> button located in the upper right of the screen. In the modal for a new panel, you need to select the dashboard that the panel is defined under, then add a command name, along with a friendly name, and define the <code class="language-plaintext highlighter-rouge">Panel ID</code>.</p> <p>The <code class="language-plaintext highlighter-rouge">Active</code> checkbox will allow the command to show up in your chat client. If the panel is marked as inactive, it will still be defined in Nautobot, but restricted from being shown in the chat client.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_grafana/panel_add.png" alt="New Panel" /></p> <blockquote> <p>NOTE: You can find the panel id by navigating to your desired panel, selecting <code class="language-plaintext highlighter-rouge">View</code>, then looking at the URL. <img src="../../../static/images/blog_posts/nautobot_chatops_grafana/panel_url.png" alt="New Panel" /></p> </blockquote> <h4 id="defining-panels-using-the-sync-method">Defining Panels Using the Sync Method</h4> <p>Alternatively, you can define a set of panels by synchronizing your Grafana panels configuration for a given dashboard to the Grafana plugin. To synchronize panels for a dashboard, within Nautobot, navigate to <code class="language-plaintext highlighter-rouge">Plugins &gt; Panels</code> and click the <code class="language-plaintext highlighter-rouge">Sync</code> button.</p> <p>This process will utilize the <a href="https://diffsync.readthedocs.io/en/latest/">DiffSync</a> library to synchronize, create, update, and delete panels in Nautobot with the Dashboard Panels that are defined in the Grafana application. Once complete, you will see all panels for a dashboard imported into Nautobot.</p> <blockquote> <p><strong>Panels are synchronized on a per-dashboard basis.</strong> <strong>All panels synchronized will be <code class="language-plaintext highlighter-rouge">INACTIVE</code> by default, you will need to set them to active to see them in Chat.</strong></p> </blockquote> <p>Once your dashboard and panels have been defined, and you activate the panels you wish to expose to the chat client, you will be able to see the available chat commands, as well as run commands to generate your panels. <img src="../../../static/images/blog_posts/nautobot_chatops_grafana/ss_top_host_timespan_P12M.png" alt="Chat Example" /></p> <h2 id="advanced-usage">Advanced Usage</h2> <p>Additional functionality can be added to the Grafana ChatOps plugin if you have variables defined on your dashboards. Panel variables can also be imported via the “Sync” functionality and associated with a panel. Then you can go in and customize how the variables behave and even enrich the ChatOps experience using Nautobot as a Source of Truth for your variables!</p> <p>To read more on the advanced usage of the Grafana ChatOps plugin with panel variables, refer to the <a href="https://github.com/nautobot/nautobot-plugin-chatops-grafana/blob/main/docs/advanced_usage.md">Advanced Usage Guide</a> in the repository.</p> <h2 id="conclusion">Conclusion</h2> <p>ChatOps has given a conduit to retrieve and respond interactively using a platform that is already in place and used for communication across almost any device, while Grafana has provided a feature-rich observability platform. With the Nautobot Grafana integration, we can now have the best of both worlds. Let us know how you’re using the Grafana ChatOps or if you have any questions or issues in the <a href="https://github.com/nautobot/nautobot-plugin-chatops-grafana/issues">GitHub repo</a>.</p> <p>-Josh Silvas</p>Josh SilvasTwo of the more intriguing topics I have heard lately that also seems to resonates with network engineers and network professionals is the insight telemetry provides, and the ease of use chat platforms such as Slack and Microsoft Teams deliver to your keyboard and fingertips. The Grafana ChatOps application is designed to provide the best of both worlds. Grafana ChatOps is a Nautobot extension used with the Nautobot ChatOps base framework to provide all the operational graphs provided by Grafana delivered via chat clients.Nautobot Chatops for Cisco ACI2021-12-07T00:00:00+00:002021-12-07T00:00:00+00:00https://blog.networktocode.com/post/nautobot-chatops-for-cisco-aci<p>We’re excited to announce the newest addition to our growing list of <a href="https://github.com/nautobot">Nautobot</a> Chatops Applications, the <a href="https://github.com/nautobot/nautobot-plugin-chatops-aci">Cisco ACI Chatops Plugin</a>! The Cisco ACI Chatops integration makes it possible to interact with the ACI controller, the <code class="language-plaintext highlighter-rouge">APIC</code> (Application Policy Infrastructure Controller), using chat commands in Slack, Mattermost, Cisco Webex, and Microsoft Teams. With this integration, network operations teams supporting Cisco ACI can use chat commands to:</p> <ul> <li>execute commands against multiple different APIC clusters in different data centers and/or regions</li> <li>register new leaf or spine switches in the fabric using chat commands</li> <li>quickly glean useful data from an APIC for informational or troubleshooting purposes</li> </ul> <p>Below, we’ll review some of the features and commands in this initial release.</p> <h2 id="multi-fabric-support">Multi-Fabric Support</h2> <p>Multi-Fabric refers to the ability to provide support for multiple APIC clusters. While there is not currently support today for the Cisco Multi-Site Orchestrator (MSO), which provides a single point of management for multiple APIC clusters, the ACI Chatops app supports configuration of as many individual APIC controllers as needed. The chat platform will prompt for the APIC to execute a command against (it’s also open source, so contributions are more than welcome!):</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/selection-menu.png" alt="APIC Selection Menu" /></p> <p>In addition, the selection dialogue can be avoided, if desired, simply by providing the APIC cluster name as the second argument to the chat command. For example, to execute against the APIC cluster called <code class="language-plaintext highlighter-rouge">ntcapic</code> in our configuration, the command would be:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/aci get-tenants ntcapic </code></pre></div></div> <blockquote> <p><code class="language-plaintext highlighter-rouge">ntcapic</code> is a friendly name assigned to our APIC cluster. Under the hood it informs the Chatops app which APIC hostname and credentials to use. The details can be found in the <a href="https://github.com/nautobot/nautobot-plugin-chatops-aci#installation">Installation Guide</a>.</p> </blockquote> <h2 id="node-registration">Node Registration</h2> <p>In Cisco ACI, when new Leaf and Spine switches are plugged into the ACI fabric for the first time, they are discovered using LLDP (Link Layer Discovery Protocol) and show up in the APIC as unregistered nodes. An administrator must then access the APIC GUI and register the new node by assigning it a name and unique ID number. The below set of chat commands could be used by network operators to view registered and pending nodes, and then register a newly discovered node in the fabric.</p> <h3 id="get-nodes">Get Nodes</h3> <p>Displays a list of all registered nodes in the fabric.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-nodes.png" alt="Get Nodes" /></p> <h3 id="get-pending-nodes">Get Pending Nodes</h3> <p>Displays a list of any unregistered nodes that have been discovered.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-pending-nodes.png" alt="Get Pending Nodes" /></p> <h3 id="register-node">Register Node</h3> <p>Register a new node in the ACI fabric.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/register-node.png" alt="Register Node" /></p> <h2 id="information-gathering">Information Gathering</h2> <p>The below chat commands can be used to retrieve and display configuration and operational details from the APIC.</p> <h3 id="display-apic-details">Display APIC Details</h3> <p>Don’t remember the hostname or IP addressing details of the APIC? Need to look up the serial number or model information? No problem, just run the <code class="language-plaintext highlighter-rouge">aci get-controllers</code> command!</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-controllers.png" alt="Get Controllers" /></p> <h3 id="display-tenants">Display Tenants</h3> <p>Get the list of tenants from an APIC using the command <code class="language-plaintext highlighter-rouge">aci get-tenants</code>.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-tenants.png" alt="Get Tenants" /></p> <h3 id="display-application-profiles">Display Application Profiles</h3> <p>Get the list of Application Profiles in a specified tenant using the <code class="language-plaintext highlighter-rouge">aci get-aps</code> command.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-aps.png" alt="Get Aps" /></p> <p>You can also specify <code class="language-plaintext highlighter-rouge">all</code> for the tenant field to get a list of all Application Profiles in the fabric across all tenants.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-aps-all.png" alt="Get All Aps" /></p> <h3 id="display-endpoint-groups-epgs">Display Endpoint Groups (EPGs)</h3> <p>Get a list of all EPGs in a specified Application Profile using the <code class="language-plaintext highlighter-rouge">aci get-epgs</code> command.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-epgs.png" alt="Get EPGs" /></p> <p>You can also specify <code class="language-plaintext highlighter-rouge">all</code> for the Application Profile selection dialogue to see EPGs for all Application Profiles in a tenant.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-epgs-all-ap.png" alt="Get EPGs All APs" /></p> <p>How about all EPGs across all tenants? Sure, just specify <code class="language-plaintext highlighter-rouge">all</code> for the Tenant dialogue…</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-epgs-all-tenants.png" alt="Get EPGs All Tenants" /></p> <h3 id="display-endpoint-group-details">Display Endpoint Group Details</h3> <p>The <code class="language-plaintext highlighter-rouge">aci get-epg-details</code> chat command provides useful information about a specified EPG, consolidating information from several API calls.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-epg-details.png" alt="Get EPG Details" /></p> <h3 id="display-vrfs">Display VRFs</h3> <p>The <code class="language-plaintext highlighter-rouge">aci get-vrfs</code> chat command displays the VRFs in use in a specified tenant.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-vrfs.png" alt="Get VRFs" /></p> <p>It is also possible to display all VRFs in the fabric by selecting <code class="language-plaintext highlighter-rouge">all</code> from the tenant selection dialogue.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-vrfs-all.png" alt="Get VRFs All" /></p> <h3 id="display-bridge-domains">Display Bridge Domains</h3> <p>The <code class="language-plaintext highlighter-rouge">aci get-bds</code> command displays Bridge Domains in a specified tenant and includes useful details from several API calls, such as the configured subnet, VRF, L2 Forwarding, and L3 Routing details.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-bds.png" alt="Get BDs" /></p> <p>You can also get a fabric-wide view of Bridge Domains by selecting <code class="language-plaintext highlighter-rouge">all</code> from the tenant selection dialogue.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-bds-all.png" alt="Get BDs All" /></p> <h3 id="display-interface-state">Display Interface State</h3> <p>The <code class="language-plaintext highlighter-rouge">aci get-interfaces</code> command can be used to quickly view interface state on a specified node.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-interfaces-all.png" alt="Get Interfaces All" /></p> <p>You can also filter for all operational or non-operational interfaces by selecting <code class="language-plaintext highlighter-rouge">up</code> or <code class="language-plaintext highlighter-rouge">down</code> from the Interface State selection dialogue.</p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-interfaces-up.png" alt="Get Interfaces Up" /></p> <p><img src="../../../static/images/blog_posts/nautobot-chatops-aci/get-interfaces-down.png" alt="Get Interfaces Down" /></p> <h2 id="wrapping-up">Wrapping Up</h2> <p>With the commands developed so far, our main focus was on providing the ability to glean useful operational details from an ACI fabric; but we could easily implement any task using the extensible API that Cisco ACI provides. What other chat commands for Cisco ACI would you find useful? Please feel free to hit us up in the comments or in our <a href="https://networktocode.slack.com">Public Slack channel</a>.</p> <p>-Matt</p>Matt MullenWe’re excited to announce the newest addition to our growing list of Nautobot Chatops Applications, the Cisco ACI Chatops Plugin! The Cisco ACI Chatops integration makes it possible to interact with the ACI controller, the APIC (Application Policy Infrastructure Controller), using chat commands in Slack, Mattermost, Cisco Webex, and Microsoft Teams. With this integration, network operations teams supporting Cisco ACI can use chat commands to:Data Modeling for Network Engineers2021-11-30T00:00:00+00:002021-11-30T00:00:00+00:00https://blog.networktocode.com/post/data-modeling-for-network-engineers<p>Structured Data, Schemas, Network as Code, and Sources of Truth—what do these all have in common? Data is central to all of them in concept and in practice. How do we work with all of that data? By modeling it of course! We’ll discuss how to start data modeling. Data modeling is not typically the focus of the conversation but is usually what you are working with, and it’s important to understand some ways to approach data modeling.</p> <p>Data models are abstractions—they detail the information and relationships we need to take into account that, when combined with assumptions, come up with the entire description of the subject. Data modeling is more of an art than a science. Coming up with a data model will almost always be an iterative approach: having too much and paring back or having too little and adding.</p> <p>Generally, there are two approaches to modeling:</p> <ul> <li>Top-down: Starting with what you want and expanding/refining</li> <li>Bottom-up: Starting with what you have and refining</li> </ul> <p>In top-down modeling, you typically try to come up with the data that you think you’ll need; likely it won’t be all or the same data that you end up with. In this case, the data modeling exercise is much like developing a new design.</p> <p>In bottom-up modeling, you will have many more points of data than your model will end up describing. The beginning of a bottom-up process would likely be a configuration itself. As you see as we begin to cover in our example, we start to peel the layers back as we make assumptions and take into account certain facts or axioms about our data, design, and configurations.</p> <h2 id="diving-into-topologies">Diving into Topologies</h2> <p>We’re going to cover more of the bottom-up process today since that’s where a lot of NetDevOps are starting: with existing networks, data, and configurations that need to be reasoned over and modeled.</p> <p>We’ll go over an example focused on modeling Devices’ Connections and what we need to model in order to derive their configurations. Consider the diagram below. It is a typical campus design where a location aggregation or distribution router aggregates all of the connections from the building and then connects to the core. This is a very common network design pattern. We’re interested in modeling the Layer 3 interfaces of Dist A in the diagram below. Generally, this modeling exercise should be applicable to most other Layer 3 interfaces in the network, but especially other Layer 3 interfaces connected to core devices on distribution switches.</p> <p><img src="../../../static/images/blog_posts/data-modeling-for-network-engineers/11_30_21_data_modeling.png" alt="Topology with core routers core-1 and core-2 connected to distribution router dist-a." /></p> <h2 id="configurations">Configurations</h2> <p>Because our design has been ruthlessly standardized, the topology above is a good example for all of our campus buildings. From the diagram, we can immediately determine that our two distribution switches should have some standard ports dedicated to certain functions: 1 uplink to the campus core, 2 cross-connects, and 2 downlinks to access switches for each distribution switch. Here’s a snippet of the configuration for dist-a.</p> <p>interface 1/1/55 no shutdown mtu 9198 qos trust none description Connection to C.Core 1 ip mtu 9198 ip address 10.10.1.2/30 arp timeout 600</p> <p>interface 1/1/56 no shutdown mtu 9198 qos trust none description Connection to C.Core 2 ip mtu 9198 ip address 10.10.1.5/30 arp timeout 600</p> <p>Keep in mind that we’re trying to get the minimum amount of data that will be able to regenerate the configuration above.</p> <p>From this configuration, we could start with a model like this:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">[{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1/1/55"</span><span class="p">,</span><span class="w"> </span><span class="nl">"shutdown"</span><span class="p">:</span><span class="w"> </span><span class="err">False</span><span class="p">,</span><span class="w"> </span><span class="nl">"lag"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"mtu"</span><span class="p">:</span><span class="w"> </span><span class="mi">9198</span><span class="p">,</span><span class="w"> </span><span class="nl">"qos-trust"</span><span class="p">:</span><span class="w"> </span><span class="s2">"none"</span><span class="p">,</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Connection to C.Core 1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"routing"</span><span class="p">:</span><span class="w"> </span><span class="err">True</span><span class="p">,</span><span class="w"> </span><span class="nl">"trunking-mode"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"allowed-vlan"</span><span class="p">:</span><span class="w"> </span><span class="s2">"all"</span><span class="p">,</span><span class="w"> </span><span class="nl">"native-vlan"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-mtu"</span><span class="p">:</span><span class="w"> </span><span class="mi">9198</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10.10.1.2/30"</span><span class="p">,</span><span class="w"> </span><span class="nl">"arp-timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">600</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1/1/56"</span><span class="p">,</span><span class="w"> </span><span class="nl">"shutdown"</span><span class="p">:</span><span class="w"> </span><span class="err">False</span><span class="p">,</span><span class="w"> </span><span class="nl">"lag"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"mtu"</span><span class="p">:</span><span class="w"> </span><span class="mi">9198</span><span class="p">,</span><span class="w"> </span><span class="nl">"qos-trust"</span><span class="p">:</span><span class="w"> </span><span class="s2">"none"</span><span class="p">,</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Connection to C.Core 2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"routing"</span><span class="p">:</span><span class="w"> </span><span class="err">True</span><span class="p">,</span><span class="w"> </span><span class="nl">"trunking-mode"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"allowed-vlan"</span><span class="p">:</span><span class="w"> </span><span class="s2">"all"</span><span class="p">,</span><span class="w"> </span><span class="nl">"native-vlan"</span><span class="p">:</span><span class="w"> </span><span class="err">None</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-mtu"</span><span class="p">:</span><span class="w"> </span><span class="mi">9198</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10.10.1.5/30"</span><span class="p">,</span><span class="w"> </span><span class="nl">"arp-timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">600</span><span class="w"> </span><span class="p">}]</span><span class="w"> </span></code></pre></div></div> <p>For data above, it is very verbose, touching on each possible configuration across most of the interfaces. This would quickly get out of hand if we need to create and manage this for every interface across all the switches across a campus. Let’s pare the model back some for our specific interface above.</p> <p>We can start with taking away properties we can assume at the point of use (in the template or when building out the data) or assumed defaults:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>If there is an IP, routing will be enabled and trunking will not. All of our configured interfaces will be enabled. The MTU will always be 9198. If there is an IP, we need an IP-MTU statement. ARP Timeout will always be 600. </code></pre></div></div> <p>With these assumptions, our model would look something like this</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1/1/55"</span><span class="p">,</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Connection to C.Core 1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10.10.1.2/30"</span><span class="p">,</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1/1/56"</span><span class="p">,</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Connection to C.Core 2"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ip-address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10.10.1.5/30"</span><span class="p">,</span><span class="w"> </span><span class="p">}]</span><span class="w"> </span></code></pre></div></div> <p>Our resulting data model above is much less verbose, making it easier to read and manage. Keep in mind the data that we will put into the model for rendering must be stored somewhere, so the fewer points of data we need for each interface the better.</p> <h2 id="usage">Usage</h2> <p>After developing the data model, we can easily input the data and use it in a template such as the interface template below. While this specific template may match a certain switch model, the data input should or could match across different models of switches.</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code>interface <span class="cp">{{</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"name"</span><span class="p">]</span> <span class="cp">}}</span> mtu 9198 <span class="cp">{%</span> <span class="k">if</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"description"</span><span class="p">]</span> <span class="o">| </span><span class="nf">length</span> <span class="o">&gt;</span> <span class="nv">1</span> <span class="cp">%}</span> description <span class="cp">{{</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"description"</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"ip_addresses"</span><span class="p">]</span> <span class="o">| </span><span class="nf">length</span> <span class="o">&gt;</span> <span class="nv">0</span> <span class="cp">%}</span> no switchport <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"ip_addresses"</span><span class="p">]</span> <span class="o">| </span><span class="nf">length</span> <span class="o">&gt;</span> <span class="nv">0</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">addr</span> <span class="ow">in</span> <span class="nv">interface</span><span class="p">[</span><span class="s2">"ip_addresses"</span><span class="p">]</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">if</span> <span class="nv">addr</span><span class="p">[</span><span class="s2">"address"</span><span class="p">]</span> <span class="ow">is</span> <span class="nb">defined</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">addr</span><span class="p">[</span><span class="s2">"address"</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> ip mtu 9198 ip arp timeout 600 <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> no shutdown </code></pre></div></div> <h2 id="process">Process</h2> <p>While developing the model, it may be helpful to keep track of the model in a spreadsheet or table. Every property of the model can be kept in a spreadsheet, to make it easier to view the model all at once. Keeping the expected variable types, whether a property is required or is optional, the source of the variable in an instance of the model, and finally an example of the property value are all helpful to explain and work with the data model.</p> <p>Here is an example of values to keep track of for the data model above.</p> <table> <thead> <tr> <th>property</th> <th>attribute type</th> <th>required</th> <th>example</th> <th>system of record</th> <th>description/notes</th> </tr> </thead> <tbody> <tr> <td>name</td> <td>string</td> <td>required</td> <td>1/1/55</td> <td>nautobot</td> <td> </td> </tr> <tr> <td>description</td> <td>string</td> <td>optional</td> <td>Connection to access sw 1</td> <td>nautobot</td> <td>determined by cable connection in nautobot</td> </tr> <tr> <td>ip_addresses</td> <td>string</td> <td>optional</td> <td>10.10.1.2/30</td> <td>nautobot</td> <td>from nautobot device interface</td> </tr> </tbody> </table> <h2 id="conclusion">Conclusion</h2> <p>We went over what data modeling is, how to get started with the data modeling process for network interfaces, how the data model could be used, and how to keep track of the model. In future blogs, we’ll go over the process for modeling other aspects of configuration and how the data model could be represented and/or derived from a Source of Truth. Thanks for reading!</p> <p>Stephen Corry</p>Stephen CorryStructured Data, Schemas, Network as Code, and Sources of Truth—what do these all have in common? Data is central to all of them in concept and in practice. How do we work with all of that data? By modeling it of course! We’ll discuss how to start data modeling. Data modeling is not typically the focus of the conversation but is usually what you are working with, and it’s important to understand some ways to approach data modeling.