Jekyll2021-05-11T19:16:50+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comIntroduction to Diffing and Syncing Data with DiffSync2021-05-11T00:00:00+00:002021-05-11T00:00:00+00:00https://blog.networktocode.com/post/intro-to-diffing-and-syncing-data-with-diffsync<p>In the world of network automation, we rarely are so fortunate as to have only a single authoritative source of data to work with. More often, we are responsible for and responsive to data in multiple distinct systems. When these systems overlap in their domains of responsibility, or in the information that they manage, we find it necessary to compare data between them and resolve any differences. Doing so manually is tedious and error-prone — what’s needed here is automation! This is what has led us at Network to Code to develop the Python library, <a href="https://pypi.org/project/diffsync/">DiffSync</a>.</p> <p>A few examples of this kind of problem:</p> <ul> <li>In a network that we manage, <a href="https://www.networktocode.com/nautobot/">Nautobot</a> stores our device inventory and IPAM information, but BGP configuration of these devices is managed by a set of YAML files stored in a Git repository. We need to identify whether there are any discrepancies between the devices represented in these two data sets. And every time there’s a new Git commit, we need to repeat this check.</li> <li>Our device inventory in our Nautobot source of truth (SoT) needs to be replicated into our IT Service Management (ITSM) platform so that devices can be associated with tickets. As devices are deployed, configured, and eventually decommissioned, both systems need to accurately reflect these changes.</li> <li>We have a legacy database system that’s slated to be decommissioned, and we need to periodically pull the latest updates in the legacy system over into the new database that’s starting to take over.</li> </ul> <p>These and countless other real-world data comparison and synchronization problems are what led us to develop DiffSync. In short, DiffSync is a generic utility library that can be used to compare (“diff”) and synchronize (“sync”) different data sets. It’s free and open source, and we’ve been using it for some time now to save development time on projects both internal and external. DiffSync is the engine underlying tools such as <a href="https://pypi.org/project/network-importer/">Network Importer</a> and the <a href="https://pypi.org/project/nautobot-netbox-importer/">Nautobot NetBox Importer</a> plugin for Nautobot.</p> <p>This will be the first in a short series of blog posts about DiffSync — in this post I’ll introduce what DiffSync is and the basics of how it works, and walk you through a short example of writing a Python script that will use DiffSync to identify, report, then resolve the differences between a pair of JSON files.</p> <h2 id="when-to-use-diffsync">When to Use DiffSync</h2> <p>DiffSync is at its most useful when you have multiple sources or sets of data to compare and/or synchronize, and especially if any of the following are true:</p> <ul> <li>If you need to repeatedly compare or synchronize the data sets as one or both change over time.</li> <li>If you need to account for not only the creation of new records, but also changes to and deletion of existing records as well.</li> <li>If various types of data in your data set naturally form a tree-like or parent-child relationship with other data.</li> <li>If the different data sets have some attributes in common and other attributes that are exclusive to one or the other.</li> </ul> <h2 id="overview-of-diffsync">Overview of DiffSync</h2> <p>DiffSync acts as an intermediate translation layer between all of the data sets you are diffing and/or syncing. In practical terms, this means that to use DiffSync, you will define a set of data models as well as the “adapters” needed to translate between each base data source and the data model. In Python terms, the adapters will be subclasses of the <code class="language-plaintext highlighter-rouge">DiffSync</code> class, and each data model class will be a subclass of the <code class="language-plaintext highlighter-rouge">DiffSyncModel</code> class.</p> <p><img src="../../../static/images/blog_posts/diffsync/diffsync_components.png" alt="Diagram showing relationship between DiffSync components" /></p> <p>Once you have used each adapter to load each data source into a collection of data model records, you can then ask DiffSync to “diff” the two data sets, and it will produce a structured representation of the difference between them. In Python, this is accomplished by calling the <code class="language-plaintext highlighter-rouge">diff_to()</code> or <code class="language-plaintext highlighter-rouge">diff_from()</code> method on one adapter and passing the other adapter as a parameter.</p> <p><img src="../../../static/images/blog_posts/diffsync/diffsync_diff_creation.png" alt="Diagram showing generation of a diff by DiffSync" /></p> <p>You can also ask DiffSync to “sync” one data set onto the other, and it will instruct your adapter as to the steps it needs to take to make sure that its data set accurately reflects the other. In Python, this is accomplished by calling the <code class="language-plaintext highlighter-rouge">sync_to()</code> or <code class="language-plaintext highlighter-rouge">sync_from()</code> method on one adapter and passing the other adapter as a parameter.</p> <p><img src="../../../static/images/blog_posts/diffsync/diffsync_sync.png" alt="Diagram showing the DiffSync sync operation" /></p> <h2 id="a-basic-example">A Basic Example</h2> <p>Let’s start with an intentionally basic example. Our “data sets” will be two JSON files, each of which contains a random subset of the numbers 1 through 50. We can generate these files with a little Python:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">json</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">random</span> <span class="o">&gt;&gt;&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">data1</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">50</span><span class="p">)</span> <span class="k">if</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">"file1.json"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_handle</span><span class="p">:</span> <span class="p">...</span> <span class="n">json</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">data1</span><span class="p">,</span> <span class="n">file_handle</span><span class="p">)</span> <span class="p">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">data2</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">50</span><span class="p">)</span> <span class="k">if</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">"file2.json"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_handle</span><span class="p">:</span> <span class="p">...</span> <span class="n">json</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">data2</span><span class="p">,</span> <span class="n">file_handle</span><span class="p">)</span> <span class="p">...</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h3 id="installing-diffsync">Installing DiffSync</h3> <p>DiffSync is available on PyPI, so all that’s needed is to create a Python virtual environment and install DiffSync into it:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python3.6 <span class="nt">-m</span> venv diffsync_virtualenv <span class="nv">$ </span><span class="nb">source </span>diffsync_virtualenv/bin/activate <span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>pip <span class="nb">install</span> <span class="nt">--upgrade</span> pip <span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>pip <span class="nb">install </span>diffsync </code></pre></div></div> <blockquote> <p>Note that DiffSync requires Python 3.6 or later!</p> </blockquote> <h3 id="defining-the-data-model">Defining the Data Model</h3> <p>When creating a new data model (subclass of <code class="language-plaintext highlighter-rouge">DiffSyncModel</code>) there are a few metadata properties, prefixed with <code class="language-plaintext highlighter-rouge">_</code>, that you may need to define for it to work properly. For this simple data model, the only two we need to be concerned about are <code class="language-plaintext highlighter-rouge">_modelname</code> (a descriptive label for this model) and <code class="language-plaintext highlighter-rouge">_identifiers</code> (a tuple of data fields that uniquely identify a single record of this model).</p> <p>Here we’re defining a model called <code class="language-plaintext highlighter-rouge">Number</code>, which has one attribute, <code class="language-plaintext highlighter-rouge">value</code>, that also serves as the unique identifier of each instance of this model.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSyncModel</span> <span class="k">class</span> <span class="nc">Number</span><span class="p">(</span><span class="n">DiffSyncModel</span><span class="p">):</span> <span class="s">"""Simple data model, storing only a number."""</span> <span class="c1"># DiffSync metadata fields </span> <span class="n">_modelname</span> <span class="o">=</span> <span class="s">"number"</span> <span class="n">_identifiers</span> <span class="o">=</span> <span class="p">(</span><span class="s">"value"</span><span class="p">,)</span> <span class="c1"># must be a tuple, not a single stand-alone value! </span> <span class="c1"># Data attributes on each instance of this model, including the above-listed identifier(s): </span> <span class="n">value</span><span class="p">:</span> <span class="nb">int</span> </code></pre></div></div> <blockquote> <p>In case you’re unfamiliar with the <code class="language-plaintext highlighter-rouge">value: int</code> syntax, this is the syntax used in Python 3.6 and later for <a href="https://www.python.org/dev/peps/pep-0526/">variable type annotations</a>. DiffSync is based on a library called <a href="https://pydantic-docs.helpmanual.io/">Pydantic</a>, which uses these type annotations to actually construct a data model that enforces the data types you’ve declared.</p> </blockquote> <h3 id="defining-the-adapter">Defining the Adapter</h3> <p>In this case, the two “data sets” we are going to compare share the same underlying “data source”, a simple JSON file. So here we actually only need to create a single adapter class that we can use for both data sets. In a more complex example involving differing databases or systems, you would need to create a separate adapter class for each one, but the same concepts apply.</p> <p>Here we’re defining an adapter called <code class="language-plaintext highlighter-rouge">NumbersJSONAdapter</code>, specifying that it knows about <code class="language-plaintext highlighter-rouge">Number</code> data records, and implementing logic for it to load these from the specified JSON file.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span> <span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span> <span class="k">class</span> <span class="nc">NumbersJSONAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="c1"># Tell DiffSync that when we refer to a model called "number", we will use the Number class </span> <span class="n">number</span> <span class="o">=</span> <span class="n">Number</span> <span class="c1"># Tell DiffSync which base/parent model(s) it should start with when diffing or syncing </span> <span class="n">top_level</span> <span class="o">=</span> <span class="p">[</span><span class="s">"number"</span><span class="p">]</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="nb">super</span><span class="p">().</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">source_file</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">source_file</span><span class="p">)</span> <span class="k">for</span> <span class="n">input_value</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span> <span class="c1"># Create a Number record representing this value </span> <span class="n">record</span> <span class="o">=</span> <span class="n">Number</span><span class="p">(</span><span class="n">value</span><span class="o">=</span><span class="n">input_value</span><span class="p">)</span> <span class="c1"># Add this record to our internal data store </span> <span class="bp">self</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">record</span><span class="p">)</span> </code></pre></div></div> <h3 id="generating-a-diff">Generating a Diff</h3> <p>Let’s put it all together into a single self-contained Python script:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># numbers_script.py </span><span class="kn">import</span> <span class="nn">json</span> <span class="kn">import</span> <span class="nn">pprint</span> <span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">DiffSync</span><span class="p">,</span> <span class="n">DiffSyncModel</span> <span class="kn">from</span> <span class="nn">diffsync.logging</span> <span class="kn">import</span> <span class="n">enable_console_logging</span> <span class="k">class</span> <span class="nc">Number</span><span class="p">(</span><span class="n">DiffSyncModel</span><span class="p">):</span> <span class="s">"""Simple data model, storing only a number."""</span> <span class="c1"># DiffSync metadata fields </span> <span class="n">_modelname</span> <span class="o">=</span> <span class="s">"number"</span> <span class="n">_identifiers</span> <span class="o">=</span> <span class="p">(</span><span class="s">"value"</span><span class="p">,)</span> <span class="c1"># must be a tuple, not a single standalone value! </span> <span class="c1"># Data attributes on each instance of this model, including the above-listed identifier(s): </span> <span class="n">value</span><span class="p">:</span> <span class="nb">int</span> <span class="k">class</span> <span class="nc">NumbersJSONAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="c1"># Tell DiffSync that when we refer to a model called "number", we will use the Number class </span> <span class="n">number</span> <span class="o">=</span> <span class="n">Number</span> <span class="c1"># Tell DiffSync which base/parent model(s) it should start with when diffing or syncing </span> <span class="n">top_level</span> <span class="o">=</span> <span class="p">[</span><span class="s">"number"</span><span class="p">]</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="nb">super</span><span class="p">().</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">source_file</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">source_file</span><span class="p">)</span> <span class="k">for</span> <span class="n">input_value</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span> <span class="c1"># Create a Number record representing this value </span> <span class="n">record</span> <span class="o">=</span> <span class="n">Number</span><span class="p">(</span><span class="n">value</span><span class="o">=</span><span class="n">input_value</span><span class="p">)</span> <span class="c1"># Add this record to our internal data store </span> <span class="bp">self</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">record</span><span class="p">)</span> <span class="k">def</span> <span class="nf">load_data</span><span class="p">():</span> <span class="s">"""Load both data sets and return the populated DiffSync adapter objects."""</span> <span class="n">data1</span> <span class="o">=</span> <span class="n">NumbersJSONAdapter</span><span class="p">(</span><span class="s">"file1.json"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"file1"</span><span class="p">)</span> <span class="n">data2</span> <span class="o">=</span> <span class="n">NumbersJSONAdapter</span><span class="p">(</span><span class="s">"file2.json"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"file2"</span><span class="p">)</span> <span class="k">return</span> <span class="p">(</span><span class="n">data1</span><span class="p">,</span> <span class="n">data2</span><span class="p">)</span> <span class="k">def</span> <span class="nf">diff_data</span><span class="p">():</span> <span class="s">"""Generate and print the diff between two data sets."""</span> <span class="n">data1</span><span class="p">,</span> <span class="n">data2</span> <span class="o">=</span> <span class="n">load_data</span><span class="p">()</span> <span class="n">diff</span> <span class="o">=</span> <span class="n">data1</span><span class="p">.</span><span class="n">diff_to</span><span class="p">(</span><span class="n">data2</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">diff</span><span class="p">.</span><span class="nb">str</span><span class="p">())</span> <span class="n">pprint</span><span class="p">.</span><span class="n">pprint</span><span class="p">(</span><span class="n">diff</span><span class="p">.</span><span class="nb">dict</span><span class="p">())</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">enable_console_logging</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="n">diff_data</span><span class="p">()</span> </code></pre></div></div> <p>If you save this script and execute it, you should get output similar to the following:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>python numbers_script.py 2021-05-03 13:32.47 <span class="o">[</span>info <span class="o">]</span> Beginning diff calculation <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:32.47 <span class="o">[</span>info <span class="o">]</span> Diff calculation <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> number number: 9 MISSING <span class="k">in </span>file2 number: 11 MISSING <span class="k">in </span>file2 number: 12 MISSING <span class="k">in </span>file2 number: 13 MISSING <span class="k">in </span>file2 number: 20 MISSING <span class="k">in </span>file2 number: 22 MISSING <span class="k">in </span>file2 number: 32 MISSING <span class="k">in </span>file2 number: 38 MISSING <span class="k">in </span>file2 number: 44 MISSING <span class="k">in </span>file2 number: 48 MISSING <span class="k">in </span>file2 number: 49 MISSING <span class="k">in </span>file2 number: 1 MISSING <span class="k">in </span>file1 number: 2 MISSING <span class="k">in </span>file1 number: 4 MISSING <span class="k">in </span>file1 number: 5 MISSING <span class="k">in </span>file1 number: 8 MISSING <span class="k">in </span>file1 number: 15 MISSING <span class="k">in </span>file1 number: 18 MISSING <span class="k">in </span>file1 number: 21 MISSING <span class="k">in </span>file1 number: 23 MISSING <span class="k">in </span>file1 number: 30 MISSING <span class="k">in </span>file1 number: 34 MISSING <span class="k">in </span>file1 number: 39 MISSING <span class="k">in </span>file1 number: 42 MISSING <span class="k">in </span>file1 number: 45 MISSING <span class="k">in </span>file1 number: 47 MISSING <span class="k">in </span>file1 <span class="o">{</span><span class="s1">'number'</span>: <span class="o">{</span><span class="s1">'1'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'11'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'12'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'13'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'15'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'18'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'2'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'20'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'21'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'22'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'23'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'30'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'32'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'34'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'38'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'39'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'4'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'42'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'44'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'45'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'47'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'48'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'49'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}</span>, <span class="s1">'5'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'8'</span>: <span class="o">{</span><span class="s1">'-'</span>: <span class="o">{}}</span>, <span class="s1">'9'</span>: <span class="o">{</span><span class="s1">'+'</span>: <span class="o">{}}}}</span> </code></pre></div></div> <blockquote> <p>The exact numbers reported will, of course, be different since we randomly generated these two files!</p> </blockquote> <p>As you can see, DiffSync has identified the numbers that differ between the two files as a DiffSync <code class="language-plaintext highlighter-rouge">Diff</code> object, which can be converted to a string representation or a dictionary representation as needed.</p> <blockquote> <p>DiffSync’s built-in logging uses <a href="https://www.structlog.org/en/stable/"><code class="language-plaintext highlighter-rouge">structlog</code></a> to generate log messages with structured data attached. By using <code class="language-plaintext highlighter-rouge">diffsync.logging.enable_console_logging()</code> in our script, we’re converting those log messages to standard Python log messages, but it’s possible in more advanced integrations to access the structured logs directly, allowing you to do various forms of log processing without needing to parse free-text log messages.</p> </blockquote> <h3 id="syncing-changes">Syncing Changes</h3> <p>For reporting and human-directed analysis, the above diff generation might be all that you need. But for automation, the next step is to be able to automatically resolve the diff and bring the two data sets into sync. Doing this as a dry run (without actually making any changes to the data sets) is as simple as calling <code class="language-plaintext highlighter-rouge">sync_to()</code> instead of <code class="language-plaintext highlighter-rouge">diff_to()</code>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># number_script.py </span> <span class="c1"># ... </span> <span class="k">def</span> <span class="nf">sync_data</span><span class="p">():</span> <span class="s">"""Sync the changes from data1 onto data2."""</span> <span class="n">data1</span><span class="p">,</span> <span class="n">data2</span> <span class="o">=</span> <span class="n">load_data</span><span class="p">()</span> <span class="n">data1</span><span class="p">.</span><span class="n">sync_to</span><span class="p">(</span><span class="n">data2</span><span class="p">)</span> <span class="c1"># Show that *within DiffSync* there is now no longer any diff between the two data sets! </span> <span class="k">print</span><span class="p">(</span><span class="n">data1</span><span class="p">.</span><span class="n">diff_to</span><span class="p">(</span><span class="n">data2</span><span class="p">).</span><span class="nb">str</span><span class="p">())</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">enable_console_logging</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="n">sync_data</span><span class="p">()</span> </code></pre></div></div> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>python numbers_script.py 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Beginning diff calculation <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Diff calculation <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Beginning <span class="nb">sync</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>9 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>11 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>12 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>13 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>20 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>22 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>32 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>38 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>44 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>48 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>create <span class="nv">diffs</span><span class="o">={</span><span class="s1">'+'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>49 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>1 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>2 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>4 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>5 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>8 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>15 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>18 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>21 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>23 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>30 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>34 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>39 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>42 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>45 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Deleted successfully <span class="o">[</span>diffsync.helpers] <span class="nv">action</span><span class="o">=</span>delete <span class="nv">diffs</span><span class="o">={</span><span class="s1">'-'</span>: <span class="o">{}}</span> <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">model</span><span class="o">=</span>number <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="nv">status</span><span class="o">=</span>success <span class="nv">unique_id</span><span class="o">=</span>47 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Sync <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Beginning diff calculation <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 13:49.35 <span class="o">[</span>info <span class="o">]</span> Diff calculation <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="o">(</span>no diffs<span class="o">)</span> </code></pre></div></div> <p>Note again that this is essentially a dry run — because we haven’t yet written any code telling DiffSync how to actually write changes back to the dataset, the “Created successfully” and “Deleted successfully” log messages <em>are referring only to successful changes within DiffSync’s own representation of the data!</em> We can see at the end that, at least within DiffSync itself, there are no longer any diffs between the two data set adapters.</p> <blockquote> <p>Note also that because DiffSync identifies the diff set before performing a sync, <strong>only the numbers that differ between the two data sets are being modified — numbers that exist in both data sets are left completely untouched!</strong> This is a key feature of DiffSync, as particularly on subsequent re-syncs between data sets, existing and fully synchronized data should not be modified unnecessarily.</p> </blockquote> <p>To make this more useful, of course, we need to write some code to actually output back to disk the changed data set. In this example, since our data set is a flat file rather than a collection of individual database records, we won’t implement individual <code class="language-plaintext highlighter-rouge">create()</code>, <code class="language-plaintext highlighter-rouge">update()</code>, and <code class="language-plaintext highlighter-rouge">delete()</code> methods on the <code class="language-plaintext highlighter-rouge">Number</code> class. We will instead implement <code class="language-plaintext highlighter-rouge">sync_complete()</code>, which is a callback function that DiffSync automatically calls at the end of a successful sync operation, providing us the opportunity to perform a bulk write of the entire, fully updated data set back to disk:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># numbers_script.py </span> <span class="kn">from</span> <span class="nn">diffsync</span> <span class="kn">import</span> <span class="n">Diff</span><span class="p">,</span> <span class="n">DiffSync</span><span class="p">,</span> <span class="n">DiffSyncModel</span><span class="p">,</span> <span class="n">DiffSyncFlags</span> <span class="c1"># ... </span> <span class="k">class</span> <span class="nc">NumbersJSONAdapter</span><span class="p">(</span><span class="n">DiffSync</span><span class="p">):</span> <span class="c1"># ... </span> <span class="k">def</span> <span class="nf">sync_complete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">:</span> <span class="n">DiffSync</span><span class="p">,</span> <span class="n">diff</span><span class="p">:</span> <span class="n">Diff</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">DiffSyncFlags</span><span class="p">.</span><span class="n">NONE</span><span class="p">,</span> <span class="n">logger</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="s">"""Callback after a sync has completed, updating the model data of this instance. Note: this callback is **only** triggered if the sync actually resulted in data changes. """</span> <span class="n">numbers</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">get_all</span><span class="p">(</span><span class="s">"number"</span><span class="p">)</span> <span class="n">data</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">([</span><span class="n">number</span><span class="p">.</span><span class="n">value</span> <span class="k">for</span> <span class="n">number</span> <span class="ow">in</span> <span class="n">numbers</span><span class="p">])</span> <span class="n">target_filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">filename</span><span class="si">}</span><span class="s">.new"</span> <span class="n">logger</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"Creating new output file"</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="n">target_filename</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">target_filename</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">target_file</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target_file</span><span class="p">)</span> <span class="n">logger</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"Output file created successfully"</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="n">target_filename</span><span class="p">)</span> </code></pre></div></div> <p>After adding the above callback function, if you run the script again, you’ll see a few more lines of output from the new logging calls you added, and can then confirm that your new data file was successfully created and is identical to <code class="language-plaintext highlighter-rouge">file1.json</code>:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>python numbers_script.py 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Beginning diff calculation <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> ... 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Sync <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Creating new output file <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">filename</span><span class="o">=</span>file2.json.new <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Output file created successfully <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">filename</span><span class="o">=</span>file2.json.new <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Beginning diff calculation <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> 2021-05-03 14:00.07 <span class="o">[</span>info <span class="o">]</span> Diff calculation <span class="nb">complete</span> <span class="o">[</span>diffsync.helpers] <span class="nv">dst</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file2"</span><span class="o">&gt;</span> <span class="nv">flags</span><span class="o">=</span>&lt;DiffSyncFlags.NONE: 0&gt; <span class="nv">src</span><span class="o">=</span>&lt;NumbersJSONAdapter <span class="s2">"file1"</span><span class="o">&gt;</span> <span class="o">(</span>no diffs<span class="o">)</span> </code></pre></div></div> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="nv">$ </span>diff file1.json file2.json.new <span class="o">(</span>diffsync_virtualenv<span class="o">)</span> <span class="err">$</span> </code></pre></div></div> <h2 id="conclusion">Conclusion</h2> <p>I hope this has been a helpful introduction to why and how to use DiffSync. In a future blog post we’ll dive into more advanced examples, including how to handle data sets consisting of multiple distinct database records or files, how to handle hierarchical or tree-like data sets, and how to handle data sets for which not all data attributes apply to all data sets.</p> <p>-Glenn</p>Glenn MatthewsIn the world of network automation, we rarely are so fortunate as to have only a single authoritative source of data to work with. More often, we are responsible for and responsive to data in multiple distinct systems. When these systems overlap in their domains of responsibility, or in the information that they manage, we find it necessary to compare data between them and resolve any differences. Doing so manually is tedious and error-prone — what’s needed here is automation! This is what has led us at Network to Code to develop the Python library, DiffSync.Installing Nautobot - A Complete Walk-Through2021-05-06T00:00:00+00:002021-05-06T00:00:00+00:00https://blog.networktocode.com/post/installing-nautobot<p>Nautobot 1.0 is here! With the release of Nautobot 1.0, we thought it’d be appropriate to post an article that walks a user through the entire Nautobot installation process.</p> <p>This post is intended for those who may be new to Nautobot, those who may be installing it for the first time, those who may not have a lot of experience with Unix, those for whom it’s helpful to see the same information in a slightly different format to assist learning, or any combination of those.</p> <p>This article follows and expands upon the <a href="https://nautobot.readthedocs.io/en/latest/installation/">Nautobot installation documentation</a>. For example, this post will not just tell you which specific commands to run. It will also explain why you are running the command and what the command is doing, hopefully giving you a better understanding of the mechanics of the installation. Since this article is written for those of basic to intermediate knowledge with Unix and Nautobot installation, more advanced users may prefer the installation docs instead of this article.</p> <p>This post will cover a basic installation, with all of Nautobot’s processes running on a single server running Ubuntu 20.04.2 LTS.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal tim@nautobot10:~$ </code></pre></div></div> <p>Each step in this process will show the command to be run, as well as an example of that command, and the resulting output from my actual install.</p> <p><code class="language-plaintext highlighter-rouge">$ this is a command</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ this is a command This is all the resulting output from the command from my actual install. Showing the terminal commands and output hopefully provides context. </code></pre></div></div> <h2 id="nautobot-os-requirements">Nautobot OS Requirements</h2> <p>Nautobot supports installation on Ubuntu 20.04+, CentOS 8.2+, or Red Hat Enterprise Linux (RHEL) 8.2+. Other POSIX-compliant systems including practically any flavor of Linux, BSD, or even macOS <em>should</em> work, but are not officially supported.</p> <p>Again, this article will document the install on a single server running Ubuntu 20.04.2.</p> <h2 id="nautobot-dependency-requirements">Nautobot Dependency Requirements</h2> <blockquote> <p>Note: This article will feature several CLI snippets showing the actual installation. Any CLI action with the <code class="language-plaintext highlighter-rouge">tim</code> user will signify a user with <code class="language-plaintext highlighter-rouge">root</code> access. Any CLI action with the <code class="language-plaintext highlighter-rouge">nautobot</code> user will signify the user specific to the Nautobot environment.</p> </blockquote> <p>Nautobot has the following minimum version requirements for its dependencies:</p> <ul> <li>Python: 3.6</li> <li>PostgreSQL: 9.6</li> <li>Redis: 4.0</li> </ul> <h3 id="optional-dependencies">Optional Dependencies</h3> <p>Nautobot can operate without the following optional packages, but will not be suitable for use in most production environments without them:</p> <ul> <li><a href="https://uwsgi-docs.readthedocs.io/en/latest/">uWSGI</a> WSGI server or equivalent</li> <li><a href="https://www.nginx.com/resources/wiki/">NGINX</a> HTTP server or equivalent</li> <li><a href="https://nautobot.readthedocs.io/en/latest/installation/external-authentication/">External authentication</a> (this post will not cover external authentication)</li> </ul> <h2 id="installation-overview">Installation Overview</h2> <p>Nautobot’s installation process was designed to be simple. It has the following four main sections:</p> <ul> <li>Installing the dependencies and infrastructure</li> <li>Installing Nautobot</li> <li>Configuring the web service and worker</li> <li>HTTP server configuration</li> </ul> <p>The next four sections will cover each of those areas in detail.</p> <h2 id="install-nautobot-dependencies-and-infrastructure">Install Nautobot Dependencies and Infrastructure</h2> <p>Nautobot must have the required packages along with an installed database infrastructure. This section will cover setting up that environment.</p> <h3 id="install-system-packages">Install System Packages</h3> <p>First, update the package sources list:</p> <p><code class="language-plaintext highlighter-rouge">$ sudo apt update -y</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo apt update -y [sudo] password for tim: Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease Hit:3 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease Hit:4 http://us.archive.ubuntu.com/ubuntu focal-security InRelease Reading package lists... Done Building dependency tree Reading state information... Done 57 packages can be upgraded. Run 'apt list --upgradable' to see them. tim@nautobot10:~$ </code></pre></div></div> <p>The following command will install the dependencies:</p> <p><code class="language-plaintext highlighter-rouge">$ sudo apt install -y git python3 python3-pip python3-venv python3-dev postgresql redis-server</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo apt install -y git python3 python3-pip python3-venv python3-dev postgresql redis-server Reading package lists... Done Building dependency tree Reading state information... Done python3 is already the newest version (3.8.2-0ubuntu2). python3 set to manually installed. git is already the newest version (1:2.25.1-1ubuntu3.1). git set to manually installed. The following additional packages will be installed: binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin libc6 libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev . . . &lt; --- snip --- &gt; . . . Processing triggers for man-db (2.9.1-1) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... tim@nautobot10:~$ </code></pre></div></div> <h3 id="database-setup">Database Setup</h3> <p>Nautobot 1.0.0 supports the PostgreSQL database. MySQL support is on the road map.</p> <h4 id="create-the-nautobot-database">Create the Nautobot Database</h4> <p>The following steps demonstrate how to create Nautobot’s database (named <code class="language-plaintext highlighter-rouge">nautobot</code>), create a user named <code class="language-plaintext highlighter-rouge">nautobot</code>, and then assign the <code class="language-plaintext highlighter-rouge">nautobot</code> user permissions to the <code class="language-plaintext highlighter-rouge">nautobot</code> database.</p> <p>Sudo to the <code class="language-plaintext highlighter-rouge">postgres</code> user (the Postgres installation creates this user) and get to the <code class="language-plaintext highlighter-rouge">postgres</code> prompt: <code class="language-plaintext highlighter-rouge">$ sudo -u postgres psql</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo -u postgres psql psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1)) Type "help" for help. postgres=# </code></pre></div></div> <p>Create the <code class="language-plaintext highlighter-rouge">nautobot</code> database:</p> <p><code class="language-plaintext highlighter-rouge">CREATE DATABASE nautobot;</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# CREATE DATABASE nautobot; CREATE DATABASE postgres=# </code></pre></div></div> <p>Create the <code class="language-plaintext highlighter-rouge">nautobot</code> user and assign a password:</p> <p><code class="language-plaintext highlighter-rouge">CREATE USER nautobot WITH PASSWORD '&lt;--don't use this password--&gt;';</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# CREATE USER nautobot WITH PASSWORD 'gipsy_danger_S45T%23}&gt;@T2[?'; CREATE ROLE postgres=# </code></pre></div></div> <blockquote> <p>WARNING: Don’t use the passwords from the examples.</p> </blockquote> <p>Grant privileges for the <code class="language-plaintext highlighter-rouge">nautobot</code> user in the <code class="language-plaintext highlighter-rouge">nautobot</code> database:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# GRANT ALL PRIVILEGES ON DATABASE nautobot to nautobot; GRANT postgres=# </code></pre></div></div> <p>Exit Postgres:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postgres=# \q tim@nautobot10:~$ </code></pre></div></div> <h4 id="verify-database-authentication-and-connection">Verify Database Authentication and Connection</h4> <p>Follow the example below to test database user <code class="language-plaintext highlighter-rouge">nautobot</code>’s authentication and connection to the <code class="language-plaintext highlighter-rouge">nautobot</code> database:</p> <p><code class="language-plaintext highlighter-rouge">$ psql --username nautobot --password --host localhost nautobot</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ psql --username nautobot --password --host localhost nautobot Password: psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1)) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. nautobot=&gt; \conninfo You are connected to database "nautobot" as user "nautobot" on host "localhost" (address "127.0.0.1") at port "5432". SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) nautobot=&gt; \q tim@nautobot10:~$ </code></pre></div></div> <blockquote> <p>TIP: If something goes wrong (such as an authentication problem) with your database, you can easily delete the <code class="language-plaintext highlighter-rouge">nautobot</code> user and database with the following procedure:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tim</span><span class="o">@</span><span class="n">nautobot10</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">sudo</span> <span class="o">-</span><span class="n">u</span> <span class="n">postgres</span> <span class="n">psql</span> <span class="n">psql</span> <span class="p">(</span><span class="mf">12.6</span> <span class="p">(</span><span class="n">Ubuntu</span> <span class="mf">12.6</span><span class="o">-</span><span class="mi">0</span><span class="n">ubuntu0</span><span class="p">.</span><span class="mf">20.04</span><span class="p">.</span><span class="mi">1</span><span class="p">))</span> <span class="n">Type</span> <span class="s">"help"</span> <span class="k">for</span> <span class="n">help</span><span class="p">.</span> <span class="n">postgres</span><span class="o">=</span><span class="c1"># DROP DATABASE nautobot; </span><span class="n">DROP</span> <span class="n">DATABASE</span> <span class="n">postgres</span><span class="o">=</span><span class="c1"># DROP USER nautobot; </span><span class="n">DROP</span> <span class="n">ROLE</span> <span class="n">postgres</span><span class="o">=</span><span class="c1"># \q </span><span class="n">tim</span><span class="o">@</span><span class="n">nautobot10</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> </code></pre></div> </div> <p>You can then recreate it using the steps outlined in the <strong>Create the Nautobot Database</strong> section above.</p> </blockquote> <h4 id="validate-redis">Validate Redis</h4> <p>Redis was already installed in the prior <code class="language-plaintext highlighter-rouge">apt install</code> action. Verify that it’s working:</p> <p><code class="language-plaintext highlighter-rouge">$ redis-cli ping</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ redis-cli ping PONG tim@nautobot10:~$ </code></pre></div></div> <h2 id="install-nautobot">Install Nautobot</h2> <h3 id="select-the-nautobot_root-directory-and-create-the-nautobot-system-user">Select the <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> Directory and Create the <code class="language-plaintext highlighter-rouge">nautobot</code> System User</h3> <p><code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> is the directory where all Nautobot-related items will be installed.</p> <blockquote> <p>NOTE: This install will use <code class="language-plaintext highlighter-rouge">/opt/nautobot</code> as <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code>, but you may choose any directory you wish.</p> </blockquote> <p>The following command:</p> <ul> <li>Creates the <code class="language-plaintext highlighter-rouge">nautobot</code> user</li> <li>Creates the <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> directory</li> <li>Sets <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> as <code class="language-plaintext highlighter-rouge">nautobot</code>’s home directory</li> </ul> <p><code class="language-plaintext highlighter-rouge">$ sudo useradd --system --shell /bin/bash --create-home --home-dir /opt/nautobot nautobot</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo useradd --system --shell /bin/bash --create-home --home-dir /opt/nautobot nautobot [sudo] password for tim: tim@nautobot10:~$ </code></pre></div></div> <h3 id="create-the-python-virtual-environment">Create the Python Virtual Environment</h3> <p>The following command creates the Python virtual environment (venv) for the <code class="language-plaintext highlighter-rouge">nautobot</code> user in <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> (<code class="language-plaintext highlighter-rouge">/opt/nautobot</code> in this example):</p> <p><code class="language-plaintext highlighter-rouge">$ sudo -u nautobot python3 -m venv /opt/nautobot</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo -u nautobot python3 -m venv /opt/nautobot tim@nautobot10:~$ </code></pre></div></div> <p>Up to this point, we’ve not actually set <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> as a variable referring to a specific directory in the environment. Here is where that actually gets done:</p> <p><code class="language-plaintext highlighter-rouge">$ echo "export NAUTOBOT_ROOT=/opt/nautobot" | sudo tee -a ~nautobot/.bashrc</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ echo "export NAUTOBOT_ROOT=/opt/nautobot" | sudo tee -a ~nautobot/.bashrc export NAUTOBOT_ROOT=/opt/nautobot tim@nautobot10:~$ </code></pre></div></div> <p>The above command updates <code class="language-plaintext highlighter-rouge">~/.bashrc</code> for <code class="language-plaintext highlighter-rouge">nautobot</code> so that any time you become <code class="language-plaintext highlighter-rouge">nautobot</code>, your <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> will be set automatically.</p> <p>If you want you can verify the change in <code class="language-plaintext highlighter-rouge">~/.bashrc</code>:</p> <p><code class="language-plaintext highlighter-rouge">$ cat /opt/nautobot/.bashrc</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ cat /opt/nautobot/.bashrc # ~/.bashrc: executed by bash(1) for non-login shells. . . . &lt; ---- BIG snip ---- &gt; . . . fi export NAUTOBOT_ROOT=/opt/nautobot tim@nautobot10:~$ </code></pre></div></div> <h3 id="sudo-to-nautobot-and-explore-the-virtual-environment">Sudo to <code class="language-plaintext highlighter-rouge">nautobot</code> and Explore the Virtual Environment</h3> <p>Nautobot must be installed by the <code class="language-plaintext highlighter-rouge">nautobot</code> user to prevent permissions problems.</p> <p><code class="language-plaintext highlighter-rouge">$ sudo -iu nautobot</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo -iu nautobot nautobot@nautobot10:~$ </code></pre></div></div> <p>Verify <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code>:</p> <p><code class="language-plaintext highlighter-rouge">$ echo $NAUTOBOT_ROOT</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ echo $NAUTOBOT_ROOT /opt/nautobot nautobot@nautobot10:~$ </code></pre></div></div> <p>Explore the virtual environment. Check the <code class="language-plaintext highlighter-rouge">$PATH</code> variable:</p> <p><code class="language-plaintext highlighter-rouge">$ echo $PATH</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ echo $PATH /opt/nautobot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin nautobot@nautobot10:~$ </code></pre></div></div> <p>Verify your <code class="language-plaintext highlighter-rouge">pip3</code>:</p> <p><code class="language-plaintext highlighter-rouge">$ which pip3</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ which pip3 /opt/nautobot/bin/pip3 nautobot@nautobot10:~$ </code></pre></div></div> <h4 id="automatic-access-to-the-virtual-environment">Automatic Access to the Virtual Environment</h4> <p>Earlier, we created the <code class="language-plaintext highlighter-rouge">nautobot</code> user’s venv using <code class="language-plaintext highlighter-rouge">sudo -u nautobot python3 -m venv /opt/nautobot</code>.</p> <p>Calling <code class="language-plaintext highlighter-rouge">venv</code> for <code class="language-plaintext highlighter-rouge">/opt/nautobot/</code> (<code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code>) created a directory structure within <code class="language-plaintext highlighter-rouge">/opt/nautobot</code>, including the <code class="language-plaintext highlighter-rouge">bin</code>, <code class="language-plaintext highlighter-rouge">include</code>, and <code class="language-plaintext highlighter-rouge">lib</code> directories therein. The <code class="language-plaintext highlighter-rouge">/opt/nautobot/bin</code> directory will hold all the packages for the venv.</p> <blockquote> <p>NOTE: There are a number of directories within <code class="language-plaintext highlighter-rouge">NAUTOBOT_ROOT</code> that get created during Nautobot installation, including <code class="language-plaintext highlighter-rouge">static</code>, <code class="language-plaintext highlighter-rouge">media</code>, <code class="language-plaintext highlighter-rouge">git</code>, and <code class="language-plaintext highlighter-rouge">jobs</code>.</p> </blockquote> <p>The <code class="language-plaintext highlighter-rouge">nautobot</code> user will automatically access the packages in the venv without need to explicitly initiate it because the first path in <code class="language-plaintext highlighter-rouge">nautobot</code>’s <code class="language-plaintext highlighter-rouge">$PATH</code> is <code class="language-plaintext highlighter-rouge">/opt/nautobot/bin</code>, which has all the packages that Nautobot requires.</p> <h3 id="prepare-the-virtual-environment">Prepare the Virtual Environment</h3> <p>Update Pip to the latest version. We also want to install the <code class="language-plaintext highlighter-rouge">wheel</code> library, telling Pip to try to install wheel packages when they are available. Installing a package in <code class="language-plaintext highlighter-rouge">wheel</code> format can significantly improve the installation time and eliminates some requirements for development libraries.</p> <p><code class="language-plaintext highlighter-rouge">$ pip3 install --upgrade pip wheel</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ pip3 install --upgrade pip wheel Collecting pip Downloading pip-21.1-py3-none-any.whl (1.5 MB) |████████████████████████████████| 1.5 MB 3.1 MB/s Collecting wheel Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB) Installing collected packages: pip, wheel Attempting uninstall: pip Found existing installation: pip 20.0.2 Uninstalling pip-20.0.2: Successfully uninstalled pip-20.0.2 Successfully installed pip-21.1 wheel-0.36.2 nautobot@nautobot10:~$ </code></pre></div></div> <h3 id="install-nautobot-1">Install Nautobot!</h3> <p>The time has come! Now install the Nautobot package as the <code class="language-plaintext highlighter-rouge">nautobot</code> user:</p> <p><code class="language-plaintext highlighter-rouge">$ pip3 install nautobot</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ pip3 install nautobot Collecting nautobot Downloading nautobot-1.0.0-py3-none-any.whl (11.1 MB) |████████████████████████████████| 11.1 MB 4.5 MB/s . . . &lt; --- snip --- &gt; Successfully installed Django-3.1.8 . . . nautobot-1.0.0 . . . urllib3-1.26.4 nautobot@nautobot10:~$ </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">nautobot-server</code> command is your single gateway to all things Nautobot. Use it to verify your version:</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server --version</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server --version 1.0.0 nautobot@nautobot10:~$ </code></pre></div></div> <h3 id="configure-nautobot">Configure Nautobot</h3> <p>Nautobot’s configurations are stored in a file called <code class="language-plaintext highlighter-rouge">nautobot_config.py</code>. Running <code class="language-plaintext highlighter-rouge">nautobot-server init</code> creates this file by default in <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT</code>:</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server init</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server init Configuration file created at '/opt/nautobot/nautobot_config.py' nautobot@nautobot10:~$ </code></pre></div></div> <p>There are a couple of changes in <code class="language-plaintext highlighter-rouge">nautobot_config.py</code> required to get Nautobot working. As the <code class="language-plaintext highlighter-rouge">nautobot</code> user, edit the file.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ vi /opt/nautobot/nautobot_config.py </code></pre></div></div> <p>The first required change is the <code class="language-plaintext highlighter-rouge">ALLOWED_HOSTS</code> value. You can set this to <code class="language-plaintext highlighter-rouge">['*']</code> for a quick start, but that is not suitable for a production environment.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># This is a list of valid fully-qualified domain names (FQDNs) for the Nautobot server. Nautobot will not permit write # access to the server via any other hostnames. The first FQDN in the list will be treated as the preferred name. # # Example: ALLOWED_HOSTS = ['nautobot.example.com', 'nautobot.internal.local'] </span><span class="n">ALLOWED_HOSTS</span> <span class="o">=</span> <span class="p">[</span><span class="s">'*'</span><span class="p">]</span> </code></pre></div></div> <blockquote> <p>WARNING: Attempting to access Nautobot via the Web UI via a URL not listed here will result in a <code class="language-plaintext highlighter-rouge">Bad Request (400)</code> return. We’ve not made it to the Web UI access part yet, but remember this for context.</p> </blockquote> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/1-bad-request.png" alt="" /></p> <p>You will also need to configure the <code class="language-plaintext highlighter-rouge">DATABASES</code> section of <code class="language-plaintext highlighter-rouge">nautobot_config.py</code>, updating the <code class="language-plaintext highlighter-rouge">NAUTOBOT_USER</code> and <code class="language-plaintext highlighter-rouge">NAUTOBOT_PASSWORD</code> settings with the values you defined in the prior <code class="language-plaintext highlighter-rouge">psql</code> command sequence. These are the minimum changes needed to run Nautobot on a local server; you can of course make additional changes as suits your particular environment.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># PostgreSQL database configuration. See the Django documentation for a complete list of available parameters: # https://docs.djangoproject.com/en/stable/ref/settings/#databases </span><span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"default"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"NAME"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_DATABASE"</span><span class="p">,</span> <span class="s">"nautobot"</span><span class="p">),</span> <span class="c1"># Database name </span> <span class="s">"USER"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_USER"</span><span class="p">,</span> <span class="s">"nautobot"</span><span class="p">),</span> <span class="c1"># Database username </span> <span class="s">"PASSWORD"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_PASSWORD"</span><span class="p">,</span> <span class="s">"gipsy_danger_S45T%23}&gt;@T2[?"</span><span class="p">),</span> <span class="c1"># Database password </span> <span class="s">"HOST"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_DB_HOST"</span><span class="p">,</span> <span class="s">"localhost"</span><span class="p">),</span> <span class="c1"># Database server </span> <span class="s">"PORT"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_DB_PORT"</span><span class="p">,</span> <span class="s">""</span><span class="p">),</span> <span class="c1"># Database port (leave blank for default) </span> <span class="s">"CONN_MAX_AGE"</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"NAUTOBOT_DB_TIMEOUT"</span><span class="p">,</span> <span class="mi">300</span><span class="p">),</span> <span class="c1"># Database timeout </span> <span class="s">"ENGINE"</span><span class="p">:</span> <span class="s">"django.db.backends.postgresql"</span><span class="p">,</span> <span class="c1"># Database driver (Postgres only supported!) </span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <h3 id="specify-optional-packages-within-local_requirementstxt">Specify Optional Packages Within <code class="language-plaintext highlighter-rouge">local_requirements.txt</code></h3> <p>All required Python packages will be installed when running <code class="language-plaintext highlighter-rouge">pip3 install nautobot</code>.</p> <p>Nautobot also supports optional packages that improve its functionality. For example, <a href="https://napalm-automation.net/">NAPALM</a> is a powerful automation tool for network elements because it abstracts away the underlying OS, allowing the user to leverage a unified API when interacting with a network element.</p> <blockquote> <p>NOTE: If you do not need to install any optional packages at this point, you can safely skip this step and proceed to <strong>Prepare the Database</strong>.</p> </blockquote> <p>Optional packages should be listed in <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT/local_requirements.txt</code>. The example below will specify <code class="language-plaintext highlighter-rouge">napalm</code> as a local requirement.</p> <p>To install NAPALM, add <code class="language-plaintext highlighter-rouge">napalm</code> to the <code class="language-plaintext highlighter-rouge">local_requirements.txt</code> file for later use.</p> <blockquote> <p>NOTE: <code class="language-plaintext highlighter-rouge">NAPALM_USERNAME</code> and <code class="language-plaintext highlighter-rouge">NAPALM_PASSWORD</code> must be configured in the <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT/nautobot_config.py</code> file for Nautobot to successfully use NAPALM.</p> </blockquote> <p><code class="language-plaintext highlighter-rouge">$ echo napalm &gt;&gt; $NAUTOBOT_ROOT/local_requirements.txt</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:/opt$ echo napalm &gt;&gt; $NAUTOBOT_ROOT/local_requirements.txt nautobot@nautobot10:/opt$ cat $NAUTOBOT_ROOT/local_requirements.txt napalm nautobot@nautobot10:/opt$ </code></pre></div></div> <h3 id="prepare-the-database">Prepare the Database</h3> <p>Nautobot’s database must be migrated prior to use. This creates the database tables and relationships:</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server migrate</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server migrate Wrapping model clean methods for custom validators failed because the ContentType table was not available or populated. This is normal during the execution of the migration command for the first time. Operations to perform: Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization Running migrations: Applying contenttypes.0001_initial... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0001_initial... OK Applying auth.0002_alter_permission_name_max_length... OK . . . &lt; --- snip --- &gt; . . . Applying extras.0004_populate_default_status_records... Model dcim.Device Adding and linking status Offline Adding and linking status Active Adding and linking status Planned Adding and linking status Staged Adding and linking status Failed Adding and linking status Inventory Adding and linking status Decommissioning . . . &lt; --- snip --- &gt; . . . Model circuits.Circuit Linking to existing status Planned Adding and linking status Provisioning Linking to existing status Active Linking to existing status Offline Adding and linking status Deprovisioning Adding and linking status Decommissioned . . . &lt; --- snip --- &gt; . . . Added 19, linked 29 status records Applying taggit.0003_taggeditem_add_unique_index... OK nautobot@nautobot10:~$ </code></pre></div></div> <blockquote> <p>NOTE: In the above output, notice the multiple database tables being created; a few are shown for context</p> </blockquote> <h3 id="create-a-superuser-and-the-static-directories">Create a Superuser and the Static Directories</h3> <p>The next step is to create a Nautobot superuser. This will be the account that you will use to log into the Nautobot Web UI for the first time.</p> <p>The superuser will be an administrative account that will allow you to create other users, set permissions, create security tokens, etc.</p> <blockquote> <p>NOTE: The email address field is not required, but be sure to use a very strong password.</p> </blockquote> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server createsuperuser</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server createsuperuser Username: tim Email address: Password: Password (again): Superuser created successfully. nautobot@nautobot10:~$ </code></pre></div></div> <p>Nautobot has several directories within <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT</code> (default location) that store static files for operations such as:</p> <ul> <li>Git repos (<code class="language-plaintext highlighter-rouge">git</code> directory)</li> <li>Custom jobs (<code class="language-plaintext highlighter-rouge">jobs</code> directory)</li> <li>Media such as images and attachments (<code class="language-plaintext highlighter-rouge">media</code> directory)</li> <li>CSS stylesheets for Web rendering (<code class="language-plaintext highlighter-rouge">static</code> directory)</li> </ul> <p>Here is a view of all the directories in <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ ls -ld $NAUTOBOT_ROOT/*/ drwxrwxr-x 3 nautobot nautobot 4096 Apr 30 19:26 /opt/nautobot/bin/ drwxrwxr-x 2 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/git/ drwxrwxr-x 2 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/include/ drwxrwxr-x 2 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/jobs/ drwxrwxr-x 3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/lib/ drwxrwxr-x 3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/lib64/ drwxrwxr-x 4 nautobot nautobot 4096 Apr 28 20:35 /opt/nautobot/media/ drwxrwxr-x 3 nautobot nautobot 4096 Apr 28 19:55 /opt/nautobot/share/ drwxrwxr-x 19 nautobot nautobot 4096 Apr 28 21:53 /opt/nautobot/static/ nautobot@nautobot10:~$ </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">collectstatic</code> command will create these directories if they don’t exist; it will also copy the required files to the <code class="language-plaintext highlighter-rouge">static</code> directory:</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server collectstatic</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server collectstatic 960 static files copied to '/opt/nautobot/static'. nautobot@nautobot10:~$ </code></pre></div></div> <h3 id="install-local-requirements">Install Local Requirements</h3> <blockquote> <p>NOTE: If you did not specify a <code class="language-plaintext highlighter-rouge">local_requirements.txt</code> file prior, you can skip to the <strong>Check Your Configuration</strong> section.</p> </blockquote> <p>A prior section detailed how to specify optional packages for install within <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT/local_requirements.txt</code>. Now it’s time to install those optional packages as the <code class="language-plaintext highlighter-rouge">nautobot</code> user.</p> <p><code class="language-plaintext highlighter-rouge">$ pip3 install -r $NAUTOBOT_ROOT/local_requirements.txt</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ pip3 install -r $NAUTOBOT_ROOT/local_requirements.txt Collecting napalm Downloading napalm-3.2.0-py2.py3-none-any.whl (230 kB) |████████████████████████████████| 230 kB 3.1 MB/s Requirement already satisfied: pyYAML in ./lib/python3.8/site-packages (from napalm-&gt;-r /opt/nautobot/local_requirements.txt (line 1)) (5.4.1) . . . &lt; --- snip --- &gt; . . . Successfully installed bcrypt-3.2.0 ciscoconfparse-1.5.30 colorama-0.4.4 dnspython-2.1.0 future-0.18.2 junos-eznc-2.6.0 lxml-4.6.3 napalm-3.2.0 ncclient-0.6.9 netmiko-3.4.0 ntc-templates-2.0.0 paramiko-2.7.2 passlib-1.7.4 pyeapi-0.8.4 pynacl-1.4.0 pyserial-3.5 scp-0.13.3 tenacity-7.0.0 textfsm-1.1.0 transitions-0.8.8 yamlordereddictloader-0.4.0 nautobot@nautobot10:~$ </code></pre></div></div> <h3 id="check-your-configuration">Check Your Configuration**</h3> <p>Nautobot uses Django’s native system check framework to validate the configuration, detect common problems, and provide hints for how to fix them.</p> <p>These checks run automatically when using <code class="language-plaintext highlighter-rouge">nautobot-server runserver</code> (which we will get to in a moment), but not when running in production using WSGI. Nevertheless, it’s good to get into the habit of running the checks before deployments!</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server check</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server check System check identified no issues (0 silenced). nautobot@nautobot10:~$ </code></pre></div></div> <blockquote> <p>** I really wanted to title this section “Check Yourself Before You Wreck Yourself”; please consider that to be the alternate title.</p> </blockquote> <h3 id="test-the-application">Test the Application</h3> <p>Now it’s time to see this installation’s Web UI for the first time! Start Nautobot’s development server on port 8000.</p> <p><code class="language-plaintext highlighter-rouge">$ nautobot-server runserver 0.0.0.0:8000 --insecure</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nautobot@nautobot10:~$ nautobot-server runserver 0.0.0.0:8000 --insecure Performing system checks... System check identified no issues (0 silenced). April 28, 2021 - 21:55:50 Django version 3.1.8, using settings 'nautobot_config' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. [28/Apr/2021 21:56:20] "GET / HTTP/1.1" 200 18684 </code></pre></div></div> <blockquote> <p>DANGER: DO NOT USE THIS SERVER IN A PRODUCTION SETTING. The development server is for development and testing purposes only. It is neither performant nor secure enough for production use.</p> </blockquote> <p>From your local host, connect to the name or IP of the server (as defined in ALLOWED_HOSTS) on port 8000; for example, http://127.0.0.1:8000/.</p> <blockquote> <p>NOTE: The URL that you use in order to reach the UI must be included in the <code class="language-plaintext highlighter-rouge">ALLOWED_HOSTS</code> list in <code class="language-plaintext highlighter-rouge">nautobot_config.py</code>; if not, you will receive a <code class="language-plaintext highlighter-rouge">Bad Request (400)</code> return.</p> </blockquote> <p>You can also reach Nautobot from a remote host. The example below shows reaching the development server UI via the remote server’s IP address.</p> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/1-1-initial-login-screen.png" alt="" /></p> <p>Here is some sample output you’ll see from the development server as you access the Web UI:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[28/Apr/2021 21:56:20] "GET /static/materialdesignicons-5.4.55/css/materialdesignicons.min.css HTTP/1.1" 200 269370 [28/Apr/2021 21:56:20] "GET /static/bootstrap-3.4.1-dist/css/bootstrap.min.css HTTP/1.1" 200 121457 [28/Apr/2021 21:56:20] "GET /static/jquery-ui-1.12.1/jquery-ui.css HTTP/1.1" 200 37326 [28/Apr/2021 21:56:20] "GET /static/select2-4.0.13/dist/css/select2.min.css HTTP/1.1" 200 14966 [28/Apr/2021 21:56:20] "GET /static/flatpickr-4.6.3/themes/light.css HTTP/1.1" 200 18996 [28/Apr/2021 21:56:20] "GET /static/select2-bootstrap-0.1.0-beta.10/select2-bootstrap.min.css HTTP/1.1" 200 16792 [28/Apr/2021 21:56:20] "GET /static/css/base.css?v1.0.0 HTTP/1.1" 200 8528 [28/Apr/2021 21:56:20] "GET /static/jquery/jquery-3.6.0.min.js HTTP/1.1" 200 89501 [28/Apr/2021 21:56:20] "GET /static/jquery-ui-1.12.1/jquery-ui.min.js HTTP/1.1" 200 253669 [28/Apr/2021 21:56:21] "GET /static/bootstrap-3.4.1-dist/js/bootstrap.min.js HTTP/1.1" 200 39680 [28/Apr/2021 21:56:21] "GET /static/clipboard.js/clipboard-2.0.6.min.js HTTP/1.1" 200 10454 [28/Apr/2021 21:56:21] "GET /static/img/nautobot_logo.svg HTTP/1.1" 200 13318 [28/Apr/2021 21:56:21] "GET /static/js/forms.js?v1.0.0 HTTP/1.1" 200 18603 [28/Apr/2021 21:56:21] "GET /static/flatpickr-4.6.3/flatpickr.min.js HTTP/1.1" 200 48518 [28/Apr/2021 21:56:21] "GET /static/select2-4.0.13/dist/js/select2.min.js HTTP/1.1" 200 70891 [28/Apr/2021 21:56:21] "GET /static/materialdesignicons-5.4.55/fonts/materialdesignicons-webfont.woff2?v=5.8.55 HTTP/1.1" 200 319984 [28/Apr/2021 21:56:21] "GET /static/img/favicon.ico HTTP/1.1" 200 15086 </code></pre></div></div> <p>Notice that the UI will be locked down until you authenticate. Click the <code class="language-plaintext highlighter-rouge">Log in</code> button on the upper right. You will see the login screen.</p> <p>Log in with the credentials you created earlier using <code class="language-plaintext highlighter-rouge">nautobot-server createsuperuser</code> :</p> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/2-initial-test-login.png" alt="" /></p> <p>When you authenticate in you’ll see more debug output at the development server’s command line:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[28/Apr/2021 21:58:17] "GET /login/?next=/ HTTP/1.1" 200 8332 [28/Apr/2021 21:58:34] "POST /login/ HTTP/1.1" 302 0 [28/Apr/2021 21:58:34] "GET / HTTP/1.1" 200 58444 </code></pre></div></div> <p>Once you authenticate, you will see an unlocked UI with no data:</p> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/3-authenticated-login.png" alt="" /></p> <p>Back in your terminal, type <code class="language-plaintext highlighter-rouge">Ctrl-C</code> to stop the development server. Now you’re ready to proceed to starting Nautobot as a system service.</p> <h2 id="configure-nautobots-web-service-and-worker">Configure Nautobot’s Web Service and Worker</h2> <p>Nautobot runs as a Web Server Gateway Interface (WSGI) application behind an HTTP server.</p> <p>Nautobot comes pre-installed with uWSGI to use as the WSGI server, but other WSGI servers should work equally as well. This demo will use the pre-installed uWSGI server, guiding you through configuring uWSGI and establishing Nautobot to run on system startup.</p> <h3 id="configure-uwsgi">Configure uWSGI</h3> <p>As the <code class="language-plaintext highlighter-rouge">nautobot</code> user, copy and paste the following into <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT/uwsgi.ini</code>:</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[uwsgi]</span> <span class="c">; The IP address (typically localhost) and port that the WSGI process should listen on </span><span class="py">socket</span> <span class="p">=</span> <span class="s">127.0.0.1:8001</span> <span class="c">; Fail to start if any parameter in the configuration file isn’t explicitly understood by uWSGI </span><span class="py">strict</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Enable master process to gracefully re-spawn and pre-fork workers </span><span class="py">master</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Allow Python app-generated threads to run </span><span class="py">enable-threads</span> <span class="p">=</span> <span class="s">true</span> <span class="c">;Try to remove all of the generated file/sockets during shutdown </span><span class="py">vacuum</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Do not use multiple interpreters, allowing only Nautobot to run </span><span class="py">single-interpreter</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Shutdown when receiving SIGTERM (default is respawn) </span><span class="py">die-on-term</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Prevents uWSGI from starting if it is unable load Nautobot (usually due to errors) </span><span class="py">need-app</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; By default, uWSGI has rather verbose logging that can be noisy </span><span class="py">disable-logging</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; Assert that critical 4xx and 5xx errors are still logged </span><span class="py">log-4xx</span> <span class="p">=</span> <span class="s">true</span> <span class="py">log-5xx</span> <span class="p">=</span> <span class="s">true</span> <span class="c">; ; Advanced settings (disabled by default) ; Customize these for your environment if and only if you need them. ; Ref: https://uwsgi-docs.readthedocs.io/en/latest/Options.html ; </span> <span class="c">; Number of uWSGI workers to spawn. This should typically be 2n+1, where n is the number of CPU cores present. ; processes = 5 </span> <span class="c">; If using subdirectory hosting e.g. example.com/nautobot, you must uncomment this line. Otherwise you'll get double paths e.g. example.com/nautobot/nautobot/. ; See: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.11.html#fixpathinfo-routing-action ; route-run = fixpathinfo: </span></code></pre></div></div> <p>This is a basic configuration that should suffice for most initial installations.</p> <h3 id="setup-systemd">Setup systemd</h3> <p>The <code class="language-plaintext highlighter-rouge">systemd</code> suite will control uWSGI as well as Nautobot’s background worker processes.</p> <h4 id="configure-nautobot-service">Configure Nautobot Service</h4> <p>With <strong><em>root</em></strong> permissions, copy and paste the following into <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nautobot.service</code>:</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[Unit]</span> <span class="py">Description</span><span class="p">=</span><span class="s">Nautobot WSGI Service</span> <span class="py">Documentation</span><span class="p">=</span><span class="s">https://nautobot.readthedocs.io/en/stable/</span> <span class="py">After</span><span class="p">=</span><span class="s">network-online.target</span> <span class="py">Wants</span><span class="p">=</span><span class="s">network-online.target</span> <span class="nn">[Service]</span> <span class="py">Type</span><span class="p">=</span><span class="s">simple</span> <span class="py">Environment</span><span class="p">=</span><span class="s">"NAUTOBOT_ROOT=/opt/nautobot"</span> <span class="py">User</span><span class="p">=</span><span class="s">nautobot</span> <span class="py">Group</span><span class="p">=</span><span class="s">nautobot</span> <span class="py">PIDFile</span><span class="p">=</span><span class="s">/var/tmp/nautobot.pid</span> <span class="py">WorkingDirectory</span><span class="p">=</span><span class="s">/opt/nautobot</span> <span class="py">ExecStart</span><span class="p">=</span><span class="s">/opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid --ini /opt/nautobot/uwsgi.ini</span> <span class="py">ExecStop</span><span class="p">=</span><span class="s">/opt/nautobot/bin/nautobot-server start --stop /var/tmp/nautobot.pid</span> <span class="py">ExecReload</span><span class="p">=</span><span class="s">/opt/nautobot/bin/nautobot-server start --reload /var/tmp/nautobot.pid</span> <span class="py">Restart</span><span class="p">=</span><span class="s">on-failure</span> <span class="py">RestartSec</span><span class="p">=</span><span class="s">30</span> <span class="py">PrivateTmp</span><span class="p">=</span><span class="s">true</span> <span class="nn">[Install]</span> <span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span> </code></pre></div></div> <blockquote> <p>NOTE: Notice the above configuration leverages the <code class="language-plaintext highlighter-rouge">nautobot-server start</code> management command that directly invokes uWSGI. The <code class="language-plaintext highlighter-rouge">nautobot-server</code> command is meant to server as the single entrypoint into the Nautobot application.</p> </blockquote> <h4 id="configure-nautobot-worker-service">Configure Nautobot Worker Service</h4> <p>To configure the Nautobot Worker, with <strong><em>root</em></strong> permissions, copy and paste the following into <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nautobot-worker.service</code>:</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[Unit]</span> <span class="py">Description</span><span class="p">=</span><span class="s">Nautobot Request Queue Worker</span> <span class="py">Documentation</span><span class="p">=</span><span class="s">https://nautobot.readthedocs.io/en/stable/</span> <span class="py">After</span><span class="p">=</span><span class="s">network-online.target</span> <span class="py">Wants</span><span class="p">=</span><span class="s">network-online.target</span> <span class="nn">[Service]</span> <span class="py">Type</span><span class="p">=</span><span class="s">simple</span> <span class="py">Environment</span><span class="p">=</span><span class="s">"NAUTOBOT_ROOT=/opt/nautobot"</span> <span class="py">User</span><span class="p">=</span><span class="s">nautobot</span> <span class="py">Group</span><span class="p">=</span><span class="s">nautobot</span> <span class="py">WorkingDirectory</span><span class="p">=</span><span class="s">/opt/nautobot</span> <span class="py">ExecStart</span><span class="p">=</span><span class="s">/opt/nautobot/bin/nautobot-server rqworker</span> <span class="py">Restart</span><span class="p">=</span><span class="s">on-failure</span> <span class="py">RestartSec</span><span class="p">=</span><span class="s">30</span> <span class="py">PrivateTmp</span><span class="p">=</span><span class="s">true</span> <span class="nn">[Install]</span> <span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span> </code></pre></div></div> <h3 id="reload-systemd-and-verify-the-nautobot-service">Reload <code class="language-plaintext highlighter-rouge">systemd</code> and Verify the Nautobot Service</h3> <p>Reload the <code class="language-plaintext highlighter-rouge">systemd</code> daemon to pick up the changes in the new services:</p> <p><code class="language-plaintext highlighter-rouge">$ sudo systemctl daemon-reload</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo systemctl daemon-reload tim@nautobot10:~$ </code></pre></div></div> <p>Now start the <code class="language-plaintext highlighter-rouge">nautobot</code> and <code class="language-plaintext highlighter-rouge">nautobot-worker</code> services and enable them to initiate at boot:</p> <p><code class="language-plaintext highlighter-rouge">$ sudo systemctl enable --now nautobot nautobot-worker</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo systemctl enable --now nautobot nautobot-worker Created symlink /etc/systemd/system/multi-user.target.wants/nautobot.service → /etc/systemd/system/nautobot.service. Created symlink /etc/systemd/system/multi-user.target.wants/nautobot-worker.service → /etc/systemd/system/nautobot-worker.service. tim@nautobot10:~$ </code></pre></div></div> <blockquote> <p>NOTE: The <code class="language-plaintext highlighter-rouge">/etc/systemd/system/multi-user.target</code> in the output above refers to the Runlevel (0-6) that the machine transits when powering on. Runlevel 0 is <code class="language-plaintext highlighter-rouge">poweroff.target</code>, Runlevel 5 is <code class="language-plaintext highlighter-rouge">graphical.target</code> (full user access with graphical display and networking). The symlinks created above start <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nautobot.service</code> and <code class="language-plaintext highlighter-rouge">/etc/systemd/system/nautobot-worker.service</code> as the machine transits Runlevel 2 (<code class="language-plaintext highlighter-rouge">multi-user.target</code>).</p> </blockquote> <p>Now use <code class="language-plaintext highlighter-rouge">systemctl status nautobot.service</code> to verify that the Nautobot service is running:</p> <p><code class="language-plaintext highlighter-rouge">$ systemctl status nautobot.service</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ systemctl status nautobot.service ● nautobot.service - Nautobot WSGI Service Loaded: loaded (/etc/systemd/system/nautobot.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-04-28 22:39:54 UTC; 1min 3s ago Docs: https://nautobot.readthedocs.io/en/stable/ Main PID: 2234541 (nautobot-server) Tasks: 2 (limit: 4585) Memory: 91.5M CGroup: /system.slice/nautobot.service ├─2234541 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid --&gt; └─2234793 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server start --pidfile /var/tmp/nautobot.pid --&gt; Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: --- Python VM already initialized --- Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: Python main interpreter initialized at 0x18df740 Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: python threads support enabled Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: your server socket listen backlog is limited to 100 connections Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: your mercy for graceful operations on workers is 60 seconds Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: mapped 145808 bytes (142 KB) for 1 cores Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: *** Operational MODE: single process *** Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x18df740&gt; Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: spawned uWSGI master process (pid: 2234541) Apr 28 22:39:57 nautobot10 nautobot-server[2234541]: spawned uWSGI worker 1 (pid: 2234793, cores: 1) </code></pre></div></div> <p>Because we like being thorough, let’s also check the Nautobot worker process:</p> <p><code class="language-plaintext highlighter-rouge">$ systemctl status nautobot-worker.service</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ systemctl status nautobot-worker.service ● nautobot-worker.service - Nautobot Request Queue Worker Loaded: loaded (/etc/systemd/system/nautobot-worker.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-05-04 22:10:11 UTC; 1min 4s ago Docs: https://nautobot.readthedocs.io/en/stable/ Main PID: 1122 (nautobot-server) Tasks: 2 (limit: 4585) Memory: 110.6M CGroup: /system.slice/nautobot-worker.service └─1122 /opt/nautobot/bin/python3 /opt/nautobot/bin/nautobot-server rqworker May 04 22:10:11 nautobot10 systemd[1]: Started Nautobot Request Queue Worker. May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Worker rq:worker:f24092394e3e4217a7bfafc300ddbac9: started, ver&gt; May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Subscribing to channel rq:pubsub:f24092394e3e4217a7bfafc300ddba&gt; May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 *** Listening on default, check_releases, custom_fields, webhoo&gt; May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: default May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: check_releases May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: custom_fields May 04 22:10:31 nautobot10 nautobot-server[1122]: 22:10:31 Cleaning registries for queue: webhooks </code></pre></div></div> <blockquote> <p>NOTE: The <code class="language-plaintext highlighter-rouge">nautobot.service</code> process is responsible for managing the WebUI and API; the <code class="language-plaintext highlighter-rouge">nautobot-worker.service</code> process subscribes to Redis Queue (RQ) and manages jobs, consumes tasks, and publishes results to the dababase.</p> </blockquote> <h3 id="troubleshooting">Troubleshooting</h3> <p>If the Nautobot service fails to start, issue the command <code class="language-plaintext highlighter-rouge">journalctl -eu nautobot.service</code> to check for log messages that may indicate the problem.</p> <p>For further troubleshooting steps, refer to <a href="https://nautobot.readthedocs.io/en/latest/installation/wsgi/#troubleshooting">the Nautobot documentation WSGI troubleshooting</a> section.</p> <h2 id="configure-http-server">Configure HTTP Server</h2> <p>We are almost there, I promise! This last section covers HTTP server configuration, which is the last step!</p> <h3 id="get-an-ssl-certificate">Get an SSL Certificate</h3> <p>To run Nautobot with HTTPS you will need a certificate, preferably one from a trusted commercial provider. To get off the ground, and depending on your organization’s policies, you may be able to use a self-signed certificate for testing and then later implement a trusted certificate.</p> <blockquote> <p>WARNING: A certificate from a trusted authority is highly recommended for production.</p> </blockquote> <p>Two files will be needed: the public certificate (<code class="language-plaintext highlighter-rouge">nautobot.crt</code>) and the private key (<code class="language-plaintext highlighter-rouge">nautobot.key</code>).</p> <p>This post will initially use a self-signed certificate for testing purposes, and will later cover the results of obtaining a trusted certificate from <a href="https://letsencrypt.org/getting-started">Let’s Encrypt</a> in an addendum at the end of the article.</p> <h4 id="self-signed-cert">Self-Signed Cert</h4> <p>The command below will generate a self-signed certificate, used for testing (root permissions required!):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ -keyout /etc/ssl/private/nautobot.key \ -out /etc/ssl/certs/nautobot.crt </code></pre></div></div> <blockquote> <p>NOTE: It is not necessary to answer the questions in the self-signed certificate generation procedure.</p> </blockquote> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ &gt; -keyout /etc/ssl/private/nautobot.key \ &gt; -out /etc/ssl/certs/nautobot.crt [sudo] password for tim: Generating a RSA private key ..........................................................................................+++++ ..........+++++ writing new private key to '/etc/ssl/private/nautobot.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:CO Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []: Email Address []: tim@nautobot10:~$ </code></pre></div></div> <h3 id="http-server-installation">HTTP Server Installation</h3> <p>Nautobot will support any HTTP server. <a href="https://nautobot.readthedocs.io/en/latest/installation/http-server/#http-server-installation">Nautobot’s documentation</a> covers setup instructions for NGINX, and this post will reprise those.</p> <h4 id="nginx-installation">NGINX Installation</h4> <p>First, install NGINX (root permissions required):</p> <p><code class="language-plaintext highlighter-rouge">$ sudo apt install -y nginx</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo apt install -y nginx Reading package lists... Done Building dependency tree Reading state information... Done . . . &lt; --- snip --- &gt; . . . Processing triggers for man-db (2.9.1-1) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... tim@nautobot10:~$ </code></pre></div></div> <h4 id="nginx-configuration">NGINX Configuration</h4> <p>Once NGINX is installed, copy and paste the following NGINX configuration into <code class="language-plaintext highlighter-rouge">/etc/nginx/sites-available/nautobot.conf</code> (<strong>root</strong> permissions required):</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">server</span> <span class="err">{</span> <span class="err">listen</span> <span class="err">443</span> <span class="err">ssl</span> <span class="err">http2</span> <span class="err">default_server</span><span class="c">; </span> <span class="err">listen</span> <span class="nn">[::]</span><span class="err">:443</span> <span class="err">ssl</span> <span class="err">http2</span> <span class="err">default_server</span><span class="c">; </span> <span class="err">server_name</span> <span class="err">_</span><span class="c">; </span> <span class="err">ssl_certificate</span> <span class="err">/etc/ssl/certs/nautobot.crt</span><span class="c">; </span> <span class="err">ssl_certificate_key</span> <span class="err">/etc/ssl/private/nautobot.key</span><span class="c">; </span> <span class="err">client_max_body_size</span> <span class="err">25m</span><span class="c">; </span> <span class="err">location</span> <span class="err">/static/</span> <span class="err">{</span> <span class="err">alias</span> <span class="err">/opt/nautobot/static/</span><span class="c">; </span> <span class="err">}</span> <span class="c"># For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`). </span> <span class="c"># Don't forget to set `FORCE_SCRIPT_NAME` in your `nautobot_config.py` to match. </span> <span class="c"># location /nautobot/ { </span> <span class="err">location</span> <span class="err">/</span> <span class="err">{</span> <span class="err">include</span> <span class="err">uwsgi_params</span><span class="c">; </span> <span class="err">uwsgi_pass</span> <span class="err">127.0.0.1:8001</span><span class="c">; </span> <span class="err">uwsgi_param</span> <span class="err">Host</span> <span class="err">$host</span><span class="c">; </span> <span class="err">uwsgi_param</span> <span class="err">X-Real-IP</span> <span class="err">$remote_addr</span><span class="c">; </span> <span class="err">uwsgi_param</span> <span class="err">X-Forwarded-For</span> <span class="err">$proxy_add_x_forwarded_for</span><span class="c">; </span> <span class="err">uwsgi_param</span> <span class="err">X-Forwarded-Proto</span> <span class="err">$http_x_forwarded_proto</span><span class="c">; </span> <span class="c"># If you want subdirectory hosting, uncomment this. The path must match </span> <span class="c"># the path of this location block (e.g. `/nautobot`). For NGINX the path </span> <span class="c"># MUST NOT end with a trailing "/". </span> <span class="c"># uwsgi_param SCRIPT_NAME /nautobot; </span> <span class="err">}</span> <span class="err">}</span> <span class="err">server</span> <span class="err">{</span> <span class="c"># Redirect HTTP traffic to HTTPS </span> <span class="err">listen</span> <span class="err">80</span> <span class="err">default_server</span><span class="c">; </span> <span class="err">listen</span> <span class="nn">[::]</span><span class="err">:80</span> <span class="err">default_server</span><span class="c">; </span> <span class="err">server_name</span> <span class="err">_</span><span class="c">; </span> <span class="err">return</span> <span class="err">301</span> <span class="err">https://$host$request_uri</span><span class="c">; </span><span class="err">}</span> </code></pre></div></div> <blockquote> <p>NOTE: This config redirects HTTP to HTTPS</p> </blockquote> <h4 id="enable-nautobot">Enable Nautobot</h4> <p>To enable the Nautobot Web UI, you’ll need to delete <code class="language-plaintext highlighter-rouge">/etc/nginx/sites-enabled/default</code> and create a symbolic link in the <code class="language-plaintext highlighter-rouge">sites-enabled</code> directory to the configuration file you just created:</p> <p>As user with root permissions:</p> <p><code class="language-plaintext highlighter-rouge">$ sudo rm -f /etc/nginx/sites-enabled/default</code></p> <p><code class="language-plaintext highlighter-rouge">$ sudo ln -s /etc/nginx/sites-available/nautobot.conf /etc/nginx/sites-enabled/nautobot.conf</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo rm -f /etc/nginx/sites-enabled/default tim@nautobot10:~$ sudo ln -s /etc/nginx/sites-available/nautobot.conf /etc/nginx/sites-enabled/nautobot.conf tim@nautobot10:~$ ls -l /etc/nginx/sites-enabled/ total 0 lrwxrwxrwx 1 root root 40 Apr 29 16:03 nautobot.conf -&gt; /etc/nginx/sites-available/nautobot.conf tim@nautobot10:~$ </code></pre></div></div> <h4 id="restart-nginx">Restart NGINX</h4> <p>Restart NGINX to use the new configuration (in <code class="language-plaintext highlighter-rouge">/etc/nginx/</code> above):</p> <p><code class="language-plaintext highlighter-rouge">$ sudo systemctl restart nginx</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ sudo systemctl restart nginx tim@nautobot10:~$ </code></pre></div></div> <p>If the restart fails, you will see some type of notification at the CLI when you run the <code class="language-plaintext highlighter-rouge">restart</code> command.</p> <p>Although we received no error message above, let’s run a check to see NGINX’s status:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tim@nautobot10:~$ systemctl status nginx.service ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2021-04-29 20:53:49 UTC; 17min ago Docs: man:nginx(8) Process: 2738959 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2738960 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2738961 (nginx) Tasks: 3 (limit: 4585) Memory: 12.0M CGroup: /system.slice/nginx.service ├─2738961 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─2738962 nginx: worker process └─2738963 nginx: worker process Apr 29 20:53:49 nautobot10 systemd[1]: Starting A high performance web server and a reverse proxy server... Apr 29 20:53:49 nautobot10 systemd[1]: Started A high performance web server and a reverse proxy server. tim@nautobot10:~$ </code></pre></div></div> <h4 id="the-home-stretch---connect-to-nautobot">The Home Stretch - Connect to Nautobot!</h4> <p>In a browser, connect to Nautobot. You should not have to specify a port.</p> <blockquote> <p>REMEMBER: You must connect using a valid URL from <code class="language-plaintext highlighter-rouge">ALLOWED_HOSTS</code> in <code class="language-plaintext highlighter-rouge">$NAUTOBOT_ROOT/nautobot_config.py</code>.</p> </blockquote> <p>If you used a self-signed certificate, your browser will likely warn you that attempting to connect to the site you just created is very dangerous and will likely result in puppies being harmed if you connect.</p> <p>Connect anyway, but not because you hate puppies.</p> <blockquote> <p>NOTE: In Google Chrome, depending upon your settings, you may have to explicitly type <em>‘thisisunsafe’</em> into the browser while seeing the warning screen below in order to connect. While doing so, you won’t see your typing anywhere in the browser.</p> </blockquote> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/4-not-private.png" alt="" /></p> <p>You should now see something like this:</p> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/5-connected-to-nautobot.png" alt="" /></p> <p>Congratulations! This completes the Nautobot installation on Ubuntu 20.04.2.</p> <h4 id="troubleshooting-1">Troubleshooting</h4> <p>If you do not see a successful result, you can find troubleshooting actions <a href="https://nautobot.readthedocs.io/en/latest/installation/http-server/#troubleshooting">here</a>, including actions for a <code class="language-plaintext highlighter-rouge">502 Bad Gateway</code> error.</p> <h2 id="wrapping-up">Wrapping Up</h2> <p>This blog covered Nautobot installation on a fresh version of Ubuntu 20.04.2. It walked you through a complete installation, providing context for each step and real output from a real installation. I hope this was helpful! If you have any comments on anything that did not work, was not clear, something else that should have been included, or any other feedback please comment below or reach out to me on <a href="http://slack.networktocode.com/">NtC’s public Slack channel</a>.</p> <p>Have a great day!</p> <p>-Tim</p> <h3 id="addendum-getting-a-trusted-certificate">Addendum: Getting a Trusted Certificate</h3> <p>If you used a self-signed certificate to get to this point, you may have questions on how to obtain a trusted commercial certificate. In a separate installation, I initially used a self-signed certificate, but went back and used <a href="https://letsencrypt.org/getting-started/">Let’s Encrypt</a> to get a trusted certificate. Although obtaining a trusted certificate is beyond the formal scope of this article, I wanted to briefly detail my experience going through that process. This section will cover the highlights.</p> <ul> <li>Go to <a href="https://letsencrypt.org/getting-started/">Let’s Encrypt</a></li> <li>Since we have shell access on the Nautobot server, select the option to use the <a href="https://certbot.eff.org/">Certbot</a> client</li> <li>Select <code class="language-plaintext highlighter-rouge">NGINX</code> software on <code class="language-plaintext highlighter-rouge">Ubuntu 20.04</code> OS, as in the picture below:</li> </ul> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/6-certbot.png" alt="" /></p> <ul> <li><a href="https://certbot.eff.org/lets-encrypt/ubuntufocal-nginx">That page above</a> had some directions to follow</li> <li>Following those directions, choose to get a certificate and have Certbot edit the <code class="language-plaintext highlighter-rouge">NGINX</code> configuration (<code class="language-plaintext highlighter-rouge">/etc/nginx/sites-available/nautobot.conf</code>) <ul> <li>The other option is simply to receive a certificate and install it yourself</li> </ul> </li> </ul> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/7-certbot-choose.png" alt="" /></p> <ul> <li>After that process, comment out the original self-signed certificate info in <code class="language-plaintext highlighter-rouge">/etc/nginx/sites-available/nautobot.conf</code></li> <li>Here is what the <code class="language-plaintext highlighter-rouge">/etc/nginx/sites-available/nautobot.conf</code> file looked like after that process: <ul> <li>Notice the sections within that say <code class="language-plaintext highlighter-rouge">managed by Certbot</code></li> </ul> </li> </ul> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># THIS IS MY ORIGINAL SELF-SIGNED CERT INFO I COMMENTED OUT #server { # listen 443 ssl http2 default_server; # listen [::]:443 ssl http2 default_server; # # server_name _; # # ssl_certificate /etc/ssl/certs/nautobot.crt; # ssl_certificate_key /etc/ssl/private/nautobot.key; # # client_max_body_size 25m; # # location /static/ { # alias /opt/nautobot/static/; # } # # # For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`). # # Don't forget to set `FORCE_SCRIPT_NAME` in your `nautobot_config.py` to match. # # location /nautobot/ { # location / { # include uwsgi_params; # uwsgi_pass 127.0.0.1:8001; # uwsgi_param Host $host; # uwsgi_param X-Real-IP $remote_addr; # uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for; # uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto; # # # If you want subdirectory hosting, uncomment this. The path must match # # the path of this location block (e.g. `/nautobot`). For NGINX the path # # MUST NOT end with a trailing "/". # # uwsgi_param SCRIPT_NAME /nautobot; # } # #} server { # Redirect HTTP traffic to HTTPS listen 80 default_server; listen [::]:80 default_server; server_name _; return 301 https://$host$request_uri; } server { listen 443 ssl http2 ; listen [::]:443 ssl http2 ; server_name fiola-nautobot.cloud.networktocode.com; # managed by Certbot ssl_certificate /etc/letsencrypt/live/fiola-nautobot.cloud.networktocode.com/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/fiola-nautobot.cloud.networktocode.com/privkey.pem; # managed by Certbot client_max_body_size 25m; location /static/ { alias /opt/nautobot/static/; } # For subdirectory hosting, you'll want to toggle this (e.g. `/nautobot/`). </code></pre></div></div> <ul> <li>Connect to Nautobot in a new browser window, note that the <code class="language-plaintext highlighter-rouge">Not Secure</code> warning in the URL window is now a lock, indicating a trusted certificate:</li> </ul> <p><img src="../../../static/images/blog_posts/nautobot_1_0_install/8-secure-site.png" alt="" /></p>Tim FiolaNautobot 1.0 is here! With the release of Nautobot 1.0, we thought it’d be appropriate to post an article that walks a user through the entire Nautobot installation process.Ansible Become Logging2021-05-04T00:00:00+00:002021-05-04T00:00:00+00:00https://blog.networktocode.com/post/ansible-become-logging<p>This blog post aims to assist Ansible users in providing documentation to security teams when using privileged escalation (become) in Ansible playbooks.</p> <h2 id="the-problem-statement">The Problem Statement</h2> <p>Ansible does not use a specific command when using privileged escalation on remote machines see <a href="https://docs.ansible.com/ansible/latest/user_guide/become.html#privilege-escalation-must-be-general">here</a> for more details. As an example, if you wanted to restart the httpd service, the command you run on the system would be <code class="language-plaintext highlighter-rouge">sudo systemctl restart httpd</code> but when Ansible executes the task, you would see something like <code class="language-plaintext highlighter-rouge">/bin/sh BECOME-SUCCESS ; /usr/bin/python /home/myuser/.ansible/tmp/ansible-tmp-abcdef/AnsiballZ_12345.py</code>. <a href="https://docs.ansible.com/ansible/latest/dev_guide/developing_program_flow_modules.html#ansiballz-framework">Here</a> is a link explaining the Ansiballz framework. This presents two concerns to most security teams. First, they cannot limit the commands used by Ansible. Second, they cannot see which commands Ansible executes. We will be addressing the second concern.</p> <h2 id="the-solution">The Solution</h2> <p>Ansible provides <a href="https://docs.ansible.com/ansible/latest/plugins/callback.html">Callback Plugins</a> that can be used when Ansible responds to various events inside a playbook. Ansible provides a small set of default callback plugins out of the box, but none that address our problem.</p> <h3 id="custom-callback-plugin">Custom Callback Plugin</h3> <p>We will be creating a custom callback plugin that hooks into the use of <code class="language-plaintext highlighter-rouge">become</code> and logs to stdout. This will not log the actual command used by Ansible but the module and arguments passed to the module. As an example, instead of <code class="language-plaintext highlighter-rouge">systemctl restart httpd</code> the log would show the <code class="language-plaintext highlighter-rouge">ansible.builtin.service</code> module with the <code class="language-plaintext highlighter-rouge">name</code> set to <code class="language-plaintext highlighter-rouge">httpd</code> and the <code class="language-plaintext highlighter-rouge">state</code> set to <code class="language-plaintext highlighter-rouge">restarted</code>.</p> <p>First we need to build out our directory structure. We are choosing the name “become” for our plugin, but you could choose your own name. By convention, the filename and plugin name should match, but they do not have to. This assumes the callback plugin will be added to an existing Ansible project:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── ansible.cfg // Ansible Configuration file. ├── example_pb.yaml // Ansible playbook. ├── callback_plugins │ └── become.py // Our Become Callback Plugin. </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>callback_plugins <span class="nb">touch </span>callback_plugins/become.py </code></pre></div></div> <p>Then add the callback plugin to your <code class="language-plaintext highlighter-rouge">ansible.cfg</code> file.</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ansible.cfg </span><span class="nn">[defaults]</span> <span class="py">callback_whitelist</span> <span class="p">=</span> <span class="s">become</span> </code></pre></div></div> <p>From there we will create a sample playbook that restarts httpd:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># example_pb.yaml</span> <span class="nn">---</span> <span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">all</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ESCALATED</span><span class="nv"> </span><span class="s">TASK:</span><span class="nv"> </span><span class="s">RESTART</span><span class="nv"> </span><span class="s">HTTPD"</span> <span class="s">ansible.builtin.service</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">httpd</span> <span class="na">state</span><span class="pi">:</span> <span class="s">restarted</span> <span class="na">become</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">NON-ESCALATED</span><span class="nv"> </span><span class="s">TASK:</span><span class="nv"> </span><span class="s">DEBUG"</span> <span class="s">ansible.builtin.debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">This</span><span class="nv"> </span><span class="s">task</span><span class="nv"> </span><span class="s">does</span><span class="nv"> </span><span class="s">not</span><span class="nv"> </span><span class="s">require</span><span class="nv"> </span><span class="s">escalation"</span> </code></pre></div></div> <p>Next we will add the basic structure to our callback plugin. Ansible provides an <a href="https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html#callback-plugins">example plugin</a> that we will modify.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># become.py # Make coding more python3-ish, this is required for contributions to Ansible </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="p">(</span><span class="n">absolute_import</span><span class="p">,</span> <span class="n">division</span><span class="p">,</span> <span class="n">print_function</span><span class="p">)</span> <span class="n">__metaclass__</span> <span class="o">=</span> <span class="nb">type</span> <span class="c1"># not only visible to ansible-doc, it also 'declares' the options the plugin requires and how to configure them. </span><span class="n">DOCUMENTATION</span> <span class="o">=</span> <span class="s">''' callback: become requirements: - whitelist in configuration short_description: When become is used, log to stdout. description: - This callback will log the become, become method, task name, action and arguments to stdout. '''</span> <span class="kn">from</span> <span class="nn">ansible.plugins.callback</span> <span class="kn">import</span> <span class="n">CallbackBase</span> <span class="k">class</span> <span class="nc">CallbackModule</span><span class="p">(</span><span class="n">CallbackBase</span><span class="p">):</span> <span class="s">""" This callback module tells you when become is used. """</span> <span class="n">CALLBACK_VERSION</span> <span class="o">=</span> <span class="mf">2.0</span> <span class="n">CALLBACK_NAME</span> <span class="o">=</span> <span class="s">'become'</span> <span class="c1"># only needed if you ship it and don't want to enable by default </span> <span class="n">CALLBACK_NEEDS_WHITELIST</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="c1"># make sure the expected objects are present, calling the base's __init__ </span> <span class="nb">super</span><span class="p">(</span><span class="n">CallbackModule</span><span class="p">,</span> <span class="bp">self</span><span class="p">).</span><span class="n">__init__</span><span class="p">()</span> </code></pre></div></div> <p>We have now added the skeleton that Ansible requires, but we have not added any new functionality to our plugin. Callback plugins listen for events that Ansible emits. Check out the <a href="https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/__init__.py">callback init file</a> in the Ansible repository for a list of all the events supported. The event that we will be looking at hooking into is the <code class="language-plaintext highlighter-rouge">v2_runner_on_start</code>, since it passes not only the task but also the host, and it executes at the start of a task. Let’s add our first callback now.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># become.py </span><span class="k">class</span> <span class="nc">CallbackModule</span><span class="p">(</span><span class="n">CallbackBase</span><span class="p">):</span> <span class="p">...</span> <span class="k">def</span> <span class="nf">v2_runner_on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="p">,</span> <span class="n">task</span><span class="p">):</span> <span class="s">""" Process the runner on start event. """</span> <span class="k">if</span> <span class="n">task</span><span class="p">.</span><span class="n">become</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Hostname: </span><span class="si">{</span><span class="n">host</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Task: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Action: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">action</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Arguments: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">args</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> </code></pre></div></div> <p>We are accessing attributes of task and host here. In this example we are showing just a few, but you can use dir(task) or dir(host) to find many others. When we execute the playbook, we should see the following:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TASK <span class="o">[</span>Gathering Facts]<span class="k">****************************</span> ok: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>ESCALATED TASK: RESTART HTTPD]<span class="k">**************</span> Hostname: my_hostname Task: ESCALATED TASK: RESTART HTTPD Action: ansible.builtin.service Arguments: <span class="o">{</span><span class="s1">'name'</span>: <span class="s1">'httpd'</span>, <span class="s1">'state'</span>: <span class="s1">'restarted'</span><span class="o">}</span> changed: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>NON-ESCALATED TASK: DEBUG] ok: <span class="o">[</span>my_hostname] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"This task does not require escalation"</span> <span class="o">}</span> </code></pre></div></div> <h3 id="supporting-variables">Supporting Variables</h3> <p>At this point, We have a callback plugin that will log to stdout the use of <code class="language-plaintext highlighter-rouge">become</code> in our playbooks. But if our playbook uses variables, we will not see those variables. Let’s modify our example to show:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># example_pb.yaml</span> <span class="nn">---</span> <span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">all</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ESCALATED</span><span class="nv"> </span><span class="s">TASK:</span><span class="nv"> </span><span class="s">RESTART</span><span class="nv"> </span><span class="s">HTTPD"</span> <span class="s">ansible.builtin.service</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">service</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">state</span><span class="pi">:</span> <span class="s">restarted</span> <span class="na">become</span><span class="pi">:</span> <span class="no">true</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">service</span><span class="pi">:</span> <span class="s">httpd</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">NON-ESCALATED</span><span class="nv"> </span><span class="s">TASK:</span><span class="nv"> </span><span class="s">DEBUG"</span> <span class="s">ansible.builtin.debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">This</span><span class="nv"> </span><span class="s">task</span><span class="nv"> </span><span class="s">does</span><span class="nv"> </span><span class="s">not</span><span class="nv"> </span><span class="s">require</span><span class="nv"> </span><span class="s">escalation"</span> </code></pre></div></div> <p>Now let’s run the playbook:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TASK <span class="o">[</span>Gathering Facts]<span class="k">****************************</span> ok: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>ESCALATED TASK: RESTART HTTPD]<span class="k">**************</span> Hostname: my_hostname Task: ESCALATED TASK: RESTART HTTPD Action: ansible.builtin.service Arguments: <span class="o">{</span><span class="s1">'name'</span>: <span class="s1">'{{ service }}'</span>, <span class="s1">'state'</span>: <span class="s1">'restarted'</span><span class="o">}</span> changed: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>NON-ESCALATED TASK: DEBUG] ok: <span class="o">[</span>my_hostname] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"This task does not require escalation"</span> <span class="o">}</span> </code></pre></div></div> <p>In order to access variables passed into a task (or from group vars), we need to add the Templar class to our callback plugin. We will need to add a few more hooks to get all of the play data and the hostvars.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># become.py </span> <span class="kn">from</span> <span class="nn">ansible.plugins.callback</span> <span class="kn">import</span> <span class="n">CallbackBase</span> <span class="kn">from</span> <span class="nn">ansible.template</span> <span class="kn">import</span> <span class="n">Templar</span> <span class="k">class</span> <span class="nc">CallbackModule</span><span class="p">(</span><span class="n">CallbackBase</span><span class="p">):</span> <span class="p">...</span> <span class="k">def</span> <span class="nf">v2_playbook_on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">playbook</span><span class="p">):</span> <span class="s">""" Initialize self.playbook. """</span> <span class="bp">self</span><span class="p">.</span><span class="n">playbook</span> <span class="o">=</span> <span class="n">playbook</span> <span class="k">def</span> <span class="nf">v2_playbook_on_play_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">play</span><span class="p">):</span> <span class="s">""" Get the variable manager of the current play from the playbook. """</span> <span class="bp">self</span><span class="p">.</span><span class="n">play</span> <span class="o">=</span> <span class="n">play</span> <span class="bp">self</span><span class="p">.</span><span class="n">vm</span> <span class="o">=</span> <span class="n">play</span><span class="p">.</span><span class="n">get_variable_manager</span><span class="p">()</span> <span class="k">def</span> <span class="nf">_all_vars</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="s">""" Load all variables for the given inputs. """</span> <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">vm</span><span class="p">.</span><span class="n">get_vars</span><span class="p">(</span> <span class="n">play</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">play</span><span class="p">,</span> <span class="n">host</span> <span class="o">=</span> <span class="n">host</span><span class="p">,</span> <span class="n">task</span> <span class="o">=</span> <span class="n">task</span> <span class="p">)</span> <span class="k">def</span> <span class="nf">v2_runner_on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="p">,</span> <span class="n">task</span><span class="p">):</span> <span class="s">""" Process the runner on start event. """</span> <span class="n">templar</span> <span class="o">=</span> <span class="n">Templar</span><span class="p">(</span><span class="n">loader</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">playbook</span><span class="p">.</span><span class="n">get_loader</span><span class="p">(),</span> <span class="n">variables</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">_vars</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="n">task</span><span class="p">))</span> <span class="k">if</span> <span class="n">task</span><span class="p">.</span><span class="n">become</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Hostname: </span><span class="si">{</span><span class="n">host</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Task: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Action: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">action</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Arguments: </span><span class="si">{</span><span class="n">templar</span><span class="p">.</span><span class="n">template</span><span class="p">(</span><span class="n">task</span><span class="p">.</span><span class="n">args</span><span class="p">,</span> <span class="n">fail_on_undefined</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> </code></pre></div></div> <p>With the above changes, running the playbook will now fill in the variables:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TASK <span class="o">[</span>Gathering Facts]<span class="k">****************************</span> ok: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>ESCALATED TASK: RESTART HTTPD]<span class="k">**************</span> Hostname: my_hostname Task: ESCALATED TASK: RESTART HTTPD Action: ansible.builtin.service Arguments: <span class="o">{</span><span class="s1">'name'</span>: <span class="s1">'httpd'</span>, <span class="s1">'state'</span>: <span class="s1">'restarted'</span><span class="o">}</span> changed: <span class="o">[</span>my_hostname] TASK <span class="o">[</span>NON-ESCALATED TASK: DEBUG] ok: <span class="o">[</span>my_hostname] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"This task does not require escalation"</span> <span class="o">}</span> </code></pre></div></div> <h3 id="the-next-step">The Next Step</h3> <p>Now that we know how to access the task attributes and variables, we could extend any of the other popular callback plugins, such as <code class="language-plaintext highlighter-rouge">syslog_json</code>, <code class="language-plaintext highlighter-rouge">Logstash</code>, or <code class="language-plaintext highlighter-rouge">Splunk</code>, to send the data directly to the security team.</p> <p>There are a few Ansible playbook options that are not covered here, such as loop or with_items, become_method, or become_user, but this should be enough to get you started. Another suggested change would be to set <code class="language-plaintext highlighter-rouge">CALLBACK_NEEDS_WHITELIST = False</code>. With this change, we would no longer need to whitelist <code class="language-plaintext highlighter-rouge">become</code> in the <code class="language-plaintext highlighter-rouge">ansible.cfg</code> file. This, paired with Ansible Tower or AWX, would allow the security team to ensure conformance to standards. For this to work, the plugin would need to be moved to the environment created by Ansible Tower or AWX. See <a href="https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html#adding-a-plugin-locally"><code class="language-plaintext highlighter-rouge">Adding a plugin locally</code></a> for more details.</p> <h2 id="conclusion">Conclusion</h2> <p>Ansible callback plugins give us the ability to provide detailed evidence of our privileged escalation to answer some of the security team’s concerns.</p> <p>Thanks for taking the time to read!</p> <p>-Stephen</p> <p>Here is the full code from <code class="language-plaintext highlighter-rouge">become.py</code>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># become.py # Make coding more python3-ish, this is required for contributions to Ansible </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="p">(</span><span class="n">absolute_import</span><span class="p">,</span> <span class="n">division</span><span class="p">,</span> <span class="n">print_function</span><span class="p">)</span> <span class="n">__metaclass__</span> <span class="o">=</span> <span class="nb">type</span> <span class="c1"># not only visible to ansible-doc, it also 'declares' the options the plugin requires and how to configure them. </span><span class="n">DOCUMENTATION</span> <span class="o">=</span> <span class="s">''' callback: become requirements: - whitelist in configuration short_description: Logs become use to stdout description: - This callback will log the become, become method, task name, action and arguments to stdout. '''</span> <span class="kn">from</span> <span class="nn">ansible.plugins.callback</span> <span class="kn">import</span> <span class="n">CallbackBase</span> <span class="kn">from</span> <span class="nn">ansible.template</span> <span class="kn">import</span> <span class="n">Templar</span> <span class="k">class</span> <span class="nc">CallbackModule</span><span class="p">(</span><span class="n">CallbackBase</span><span class="p">):</span> <span class="s">""" This callback module tells you when become is used. """</span> <span class="n">CALLBACK_VERSION</span> <span class="o">=</span> <span class="mf">2.0</span> <span class="n">CALLBACK_NAME</span> <span class="o">=</span> <span class="s">'become'</span> <span class="c1"># only needed if you ship it and don't want to enable by default </span> <span class="n">CALLBACK_NEEDS_WHITELIST</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="c1"># make sure the expected objects are present, calling the base's __init__ </span> <span class="nb">super</span><span class="p">(</span><span class="n">CallbackModule</span><span class="p">,</span> <span class="bp">self</span><span class="p">).</span><span class="n">__init__</span><span class="p">()</span> <span class="k">def</span> <span class="nf">v2_playbook_on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">playbook</span><span class="p">):</span> <span class="s">""" Initialize self.playbook. """</span> <span class="bp">self</span><span class="p">.</span><span class="n">playbook</span> <span class="o">=</span> <span class="n">playbook</span> <span class="k">def</span> <span class="nf">v2_playbook_on_play_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">play</span><span class="p">):</span> <span class="s">""" Get the variable manager of the current play from the playbook. """</span> <span class="bp">self</span><span class="p">.</span><span class="n">play</span> <span class="o">=</span> <span class="n">play</span> <span class="bp">self</span><span class="p">.</span><span class="n">vm</span> <span class="o">=</span> <span class="n">play</span><span class="p">.</span><span class="n">get_variable_manager</span><span class="p">()</span> <span class="k">def</span> <span class="nf">_all_vars</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="s">""" Load all variables for the given inputs. """</span> <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">vm</span><span class="p">.</span><span class="n">get_vars</span><span class="p">(</span> <span class="n">play</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">play</span><span class="p">,</span> <span class="n">host</span> <span class="o">=</span> <span class="n">host</span><span class="p">,</span> <span class="n">task</span> <span class="o">=</span> <span class="n">task</span> <span class="p">)</span> <span class="k">def</span> <span class="nf">v2_runner_on_start</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="p">,</span> <span class="n">task</span><span class="p">):</span> <span class="s">""" Process the runner on start event. """</span> <span class="n">templar</span> <span class="o">=</span> <span class="n">Templar</span><span class="p">(</span><span class="n">loader</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">playbook</span><span class="p">.</span><span class="n">get_loader</span><span class="p">(),</span> <span class="n">variables</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">_vars</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="n">task</span><span class="p">))</span> <span class="k">if</span> <span class="n">task</span><span class="p">.</span><span class="n">become</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Hostname: </span><span class="si">{</span><span class="n">host</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Task: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Action: </span><span class="si">{</span><span class="n">task</span><span class="p">.</span><span class="n">action</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Arguments: </span><span class="si">{</span><span class="n">templar</span><span class="p">.</span><span class="n">template</span><span class="p">(</span><span class="n">task</span><span class="p">.</span><span class="n">args</span><span class="p">,</span> <span class="n">fail_on_undefined</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> </code></pre></div></div> <h2 id="references">References</h2> <ul> <li><a href="https://docs.ansible.com/ansible/latest/plugins/callback.html">https://docs.ansible.com/ansible/latest/plugins/callback.html</a></li> <li><a href="https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html">https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html</a></li> <li><a href="https://stackoverflow.com/a/46687716">Stack Overflow</a></li> <li><a href="https://docs.ansible.com/ansible/latest/dev_guide/developing_program_flow_modules.html#ansiballz-framework">Ansiballz Framework</a></li> </ul>Stephen KielyThis blog post aims to assist Ansible users in providing documentation to security teams when using privileged escalation (become) in Ansible playbooks.Nautobot GraphQL Requests via Postman and Python2021-04-29T00:00:00+00:002021-04-29T00:00:00+00:00https://blog.networktocode.com/post/nautobot-graphql-requests-via-postman-and-python<p>GraphQL is a powerful query tool because it allows efficient queries for specific information that can span multiple resources that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.</p> <p>This article builds on the prior articles in this series, which describe how to <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">craft GraphQL queries in Nautobot’s <em>GraphiQL</em> interface</a> and how to use <a href="https://blog.networktocode.com/post/graphql-aliasing-with-nautobot/">GraphQL <em>aliasing</em></a> to customize the returned key names and do multiple queries in a single GraphQL request.</p> <p>The <a href="https://blog.networktocode.com/post/programmatic-nautobot-graphql-requests-via-pynautobot/">previous post in this series</a> demonstrated how to leverage the Pynautobot Python package for programmatic GraphQL queries, showing a contrasting way (from the Postman-centric way described in the post below) to construct programmatic GraphQL queries in Python against Nautobot. The Pynautobot Python path provides a very clean, customized way to programmatically run simple or sophisticated GraphQL requests against Nautobot. The methodology described in this post examines a more general way to go about that, but at the expense of additional steps. Depending on your use case, one way or the other may be preferable.</p> <p>This post will demonstrate three different techniques for configuring token authentication within Postman and two different data formats in the request body.</p> <blockquote> <p>This post uses Postman version 8.0.6.</p> </blockquote> <p>The examples in this post all use the <a href="https://demo.nautobot.com/">public Nautobot demo site</a> and the <a href="https://www.postman.com/downloads/">free Postman app</a>. Readers are encouraged to follow along.</p> <h2 id="authentication">Authentication</h2> <p>Security tokens are typically required for programmatic access to Nautobot’s data. The following sections cover how to obtain a token and how to leverage it within Postman to craft GraphQL API calls.</p> <p>Nautobot security tokens are created in the Web UI. To view your token(s), navigate to the API Tokens page under your user profile. If there is no token present, create one or get the necessary permissions to do so.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/1-api-token.png" alt="" /></p> <h2 id="setting-up-postman-token-authentication">Setting Up Postman Token Authentication</h2> <p>There are multiple ways to authenticate your Nautobot API queries in Postman:</p> <ul> <li>Directly inserting the token information in the request header</li> <li>The <em>Authentication</em> tab for the request</li> <li>The <em>Authorization</em> tab for the collection</li> </ul> <h3 id="authentication-option-1-editing-the-header">Authentication Option 1: Editing the Header</h3> <p>Inserting the token authentication information directly in the header allows individual queries to authenticate, but the user must manually populate the header with the authentication info on each new query.</p> <p>To start a query that uses this method within Postman, create a new Postman collection:</p> <ul> <li>Click on the ‘+’ sign on the upper left of the app.</li> <li>Name the collection by clicking on the pencil/edit button by the default <em>New Collection</em> name.</li> </ul> <blockquote> <p>NOTE: A Postman <em>Collection</em> is a group of saved requests, oftentimes with a common theme, workflow, or function</p> </blockquote> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/2-collection.png" alt="" /></p> <p>Name the collection <em>Authentication Testing</em>.</p> <p>Create a new API request by clicking on the ‘+’ sign to the right of the new collection tab.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/3-auth-testing.png" alt="" /></p> <p>Before we create the actual request, we’ll edit the header by adding the authentication/token info:</p> <ol> <li>Go to the <em>Headers</em> tab</li> <li>View the auto-generated headers (click on the eyeball)</li> <li>In the next available row, type <code class="language-plaintext highlighter-rouge">Authorization</code> in the <em>Key</em> column (it should allow you to auto-complete)</li> <li>In the <em>Value</em> column, type in the word <code class="language-plaintext highlighter-rouge">Token&lt;space&gt;&lt;key&gt;</code> <ul> <li>In this example I entered <code class="language-plaintext highlighter-rouge">Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</code></li> </ul> </li> </ol> <blockquote> <p>The Nautobot public sandbox key is <code class="language-plaintext highlighter-rouge">aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</code></p> </blockquote> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/4-edit-header.png" alt="" /></p> <p>Your <em>Authorization</em> portion is now complete.</p> <h4 id="query-via-graphql-in-body">Query via GraphQL in Body</h4> <p>The next steps will guide you through crafting the rest of the query. This query will retrieve the name of each device along with the device’s site name.</p> <p>This example will use the <em>GraphQL</em> format in the body.</p> <ol> <li>Change the request type to <em>POST</em></li> <li>Enter the Nautobot server + GraphQL endpoint URL <code class="language-plaintext highlighter-rouge">&lt;server&gt;/api/graphql/</code></li> <li>Go to the <em>Body</em> section</li> <li>Select <em>GraphQL</em></li> <li>Input the specific query shown below</li> </ol> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">devices</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="n">site</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/5-graphql-body.png" alt="" /></p> <p>Click the <strong>Send</strong> button and wait for the return data. You should see output similar to the picture below if you are using the public-facing Nautobot demo site.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/6-returned-data.png" alt="" /></p> <p>Save the request in the collection.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/7-save-request.png" alt="" /></p> <h3 id="authentication-option-2-the-authorization-tab">Authentication Option 2: The Authorization Tab</h3> <p>This section will show an example of authentication in the request’s <em>Authorization</em> tab. This example will reside in the same <em>Authentication Testing</em> collection.</p> <p>To create the new request:</p> <ol> <li>Right-click on the collection name</li> <li>Select <em>Add Request</em></li> <li>Name the request <em>bkk devices; uses Authorization Tab</em></li> </ol> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/8-add-request.png" alt="" /></p> <p>Configure the <em>Authorization</em> tab first, using the settings shown below; the <code class="language-plaintext highlighter-rouge">Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</code> entry is the same as in the prior section.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/9-edit-auth-tab.png" alt="" /></p> <p>Go to the <em>Headers</em> tab. If the hidden headers are not already being shown, click on the hidden headers eyeball to view the hidden headers. The <em>Authorization</em> key-value pair should be visible now. This was auto-generated from info in the <em>Authorization</em> tab.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/10-show-headers.png" alt="" /></p> <h4 id="query-via-raw-data">Query via Raw Data</h4> <p>Finish out the query, but this time use <em>raw</em> in the <em>Body</em> section. Be sure to specify the format as <em>JSON</em>.</p> <p>This query will return the device names in the <code class="language-plaintext highlighter-rouge">bkk</code> site, along with each device’s rack name and device role name. Be sure to escape the double-quotes in the <strong>site</strong> query parameter as shown below.</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">query</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">query</span><span class="w"> </span><span class="p">{</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="err">\</span><span class="s2">"bkk\") {name device_role { name } rack { name } } }"</span><span class="w"> </span><span class="err">}</span><span class="w"> </span></code></pre></div></div> <blockquote> <p>NOTE: Raw format is really cumbersome to work with, in part because of the requirement to escape quotes</p> </blockquote> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/11-raw-body-query.png" alt="" /></p> <blockquote> <p>Notice the potentially tedious nature of crafting the query above using the <em>raw data</em> option. It may be best to use the <em>GraphQL</em> format in the body for more complex queries.</p> </blockquote> <p>If you check the <em>Headers</em> tab again, you’ll notice that changing the body format to <em>JSON</em> sets the <code class="language-plaintext highlighter-rouge">Content-Type</code> value to <code class="language-plaintext highlighter-rouge">application/json</code>.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/12-updated-headers.png" alt="" /></p> <h3 id="authentication-option-3-via-the-collection-environment">Authentication Option 3: Via the Collection Environment</h3> <p>Within a Postman collection, it is more practical to configure authentication in the collection’s environment instead of configuring authentication for each request. Configuring authentication for the entire collection lets the user configure authentication just once.</p> <p>This next section will describe how to set authentication at the collection level.</p> <p>To the right of the collection name there are 3 dots (1); click on them and select <em>Edit</em> (2).</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/13-collection-auth.png" alt="" /></p> <p>From here</p> <ol> <li>Select the <em>Authorization</em> tab</li> <li>Select <em>API Key</em> for Type</li> <li>Set Key to <em>Authorization</em></li> <li>Set Value to <code class="language-plaintext highlighter-rouge">Token&lt;space&gt;&lt;key&gt;</code></li> <li>Save</li> </ol> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/14-collection-auth-config.png" alt="" /></p> <p>Any added requests in this collection can now use the <em>Inherit auth from parent</em> setting.</p> <p>To configure a request to use the collection’s authentication:</p> <ol> <li>Select the <em>bkk devices</em> request that uses the <em>Authorization</em> tab (for example)</li> <li>Select the <em>Authorization</em> tab within the request</li> <li>Switch the Type to <em>Inherit auth from parent</em></li> <li>Click <em>Send</em></li> </ol> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/15-auth-from-parent.png" alt="" /></p> <p>Save this updated request if you wish.</p> <h2 id="converting-postman-queries-to-python">Converting Postman Queries to Python</h2> <p>Postman has helped us construct our GraphQL requests. This next section describes how to leverage those requests programmatically in Python.</p> <p>This section will continue on from where we left off above, with the <em>bkk devices</em> request.</p> <p>Postman has a very useful capability to export your crafted requests into a code format of your choice. On the right-hand side of the app, select the <em>Code snippet</em> button, marked as <code class="language-plaintext highlighter-rouge">&lt;/&gt;</code>.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/16-code-export-button.png" alt="" /></p> <p>Within the search box that appears you can type <em>Python</em> or select from any of the languages that appear in the dropdown menu. This example will use the Python <em>requests</em> library.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/17-select-python-requests.png" alt="" /></p> <p>Selecting the <em>Python - Requests</em> option, you will now see Python code that will programmatically make the request using Python’s <em>requests</em> library and return the response.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/18-python-code.png" alt="" /></p> <p>If cookie information appears in the code and you do not wish this to be present, you can manually delete it from the code or prevent it from being generated by clicking on <em>Cookies</em> (located directly beneath the <em>Send</em> button) to open the Cookie Manager; delete any cookies configured and then regenerate the Python code.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-postman/19-cookies.png" alt="" /></p> <p>This Python code snippet can be added to any script or simply inserted into a Python interpreter:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~ % python3 Python 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. &gt;&gt;&gt; import requests &gt;&gt;&gt; &gt;&gt;&gt; url = "https://demo.nautobot.com/api/graphql/" &gt;&gt;&gt; &gt;&gt;&gt; payload="{\n \"query\": \"query {devices(site:\\\"bkk\\\") {name device_role { name } rack { name } } }\"\n}" &gt;&gt;&gt; headers = { ... 'Authorization': 'Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', ... 'Content-Type': 'application/json' ... } &gt;&gt;&gt; &gt;&gt;&gt; response = requests.request("POST", url, headers=headers, data=payload) &gt;&gt;&gt; &gt;&gt;&gt; print(response.text) {"data":{"devices":[{"name":"bkk-edge-01","device_role":{"name":"edge"},"rack":{"name":"bkk-101"}},{"name":"bkk-edge-02","device_role":{"name":"edge"},"rack":{"name":"bkk-102"}},{"name":"bkk-leaf-01","device_role":{"name":"leaf"},"rack":{"name":"bkk-101"}},{"name":"bkk-leaf-02","device_role":{"name":"leaf"},"rack":{"name":"bkk-102"}},{"name":"bkk-leaf-03","device_role":{"name":"leaf"},"rack":{"name":"bkk-103"}},{"name":"bkk-leaf-04","device_role":{"name":"leaf"},"rack":{"name":"bkk-104"}},{"name":"bkk-leaf-05","device_role":{"name":"leaf"},"rack":{"name":"bkk-105"}},{"name":"bkk-leaf-06","device_role":{"name":"leaf"},"rack":{"name":"bkk-106"}},{"name":"bkk-leaf-07","device_role":{"name":"leaf"},"rack":{"name":"bkk-107"}},{"name":"bkk-leaf-08","device_role":{"name":"leaf"},"rack":{"name":"bkk-108"}}]}} &gt;&gt;&gt; </code></pre></div></div> <blockquote> <p>NOTE: Being that there are multiple ways to use the <em>requests</em> library, you can also use the more concise <code class="language-plaintext highlighter-rouge">requests.post</code> method: <code class="language-plaintext highlighter-rouge">response = requests.post(url, headers=headers, data=payload)</code>.</p> </blockquote> <p>Here is a Python script that uses the <code class="language-plaintext highlighter-rouge">requests.post</code> method. The script executes the GraphQL request, prints the text response, and then pretty prints the returned json data:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import json import requests from pprint import pprint print("Querying Nautobot via graphQL API call.") print() url = "https://demo.nautobot.com/api/graphql/" print("url is: {}".format(url)) print() payload="{\n \"query\": \"query {devices(site:\\\"bkk\\\") {name device_role { name } rack { name } } }\"\n}" print("payload is: {}".format(payload)) print() headers = { 'Authorization': 'Token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'Content-Type': 'application/json' } print("headers is: {}".format(headers)) print() response = requests.post(url, headers=headers, data=payload) response_data = response.json() print("Here is the response data in json:") pprint(response_data) </code></pre></div></div> <p>Note the tedious nature of crafting <code class="language-plaintext highlighter-rouge">payload</code> variable in the script above - it’s fairly complex with all the escaping backslashes. Luckily, the Postman app will create the object in the code for us, whether we use <em>GraphQL</em> or <em>raw</em> for the <strong>Body</strong> format.</p> <h2 id="wrapping-up">Wrapping Up</h2> <p>To fully leverage GraphQL’s efficiency, it must be used programmatically. Prior posts in this series demonstrated <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">using Nautobot’s <em>GraphiQL</em> interface</a> to craft GraphQL queries and <a href="https://blog.networktocode.com/post/graphql-aliasing-with-nautobot/">how to use GraphQL aliasing</a>. This post builds on that by showing how to convert those GraphQL queries into remote requests in Postman and going on to export those requests to Python code for programmatic use. Also be sure to check out the prior post on <a href="https://blog.networktocode.com/post/programmatic-nautobot-graphql-requests-via-pynautobot/">using programmatic GraphQL requests with Nautobot via the <code class="language-plaintext highlighter-rouge">pynautobot</code> package</a> and how that stacks up against the methodology in this post. Generally, the <code class="language-plaintext highlighter-rouge">pynautobot</code> path has fewer moving parts and offers customized APIs for Nautobot that allow sophisticated GraphQL queries. The Postman+requests path offers a more general framework but with more steps.</p>Tim FiolaGraphQL is a powerful query tool because it allows efficient queries for specific information that can span multiple resources that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.Cisco NSO Development Environment in Docker2021-04-27T00:00:00+00:002021-04-27T00:00:00+00:00https://blog.networktocode.com/post/nso-dev-env-in-docker<p>Developing services for Cisco NSO requires a development environment, which can be set up locally on your laptop or on a dedicated server. When a developer starts working on an NSO service, it has to install a correct NSO version, all required NEDs, and dependencies. This is where Docker can come in handy. It is a tool to make packaging easy, which means that you can pre-build a Docker image, which contains NSO, NEDs, and all dependencies.</p> <p>In this blog post, I will show you how to create an NSO Docker image that can be used for development. There are a couple of advantages of using Docker containers for development. The most important is that everything can be packaged together in a single unit.</p> <p>To build a Docker image, Docker requires a Dockerfile, where you put build instructions to create an image. I can create <code class="language-plaintext highlighter-rouge">Dockerfile</code> from scratch. But there is a better way. <a href="https://github.com/NSO-developer/nso-docker">NSO in Docker</a> from Cisco is a repository that contains scripts and files to create a base NSO Docker image. Using the NSO in Docker project will produce a base image, which can be extended with additional packages.</p> <p>This blog post is divided in three sections:</p> <ul> <li>Prepare a base image</li> <li>Extend the base image</li> <li>Run a Docker container</li> </ul> <p><br /></p> <h2 id="prepare-a-base-image">Prepare a Base Image</h2> <p>The first step in the process is to build a base image, which is a bare NSO installation that contains NSO only. The NSO in Docker project is used in this step.</p> <h3 id="clone-the-repository">Clone the Repository</h3> <p>The NSO in Docker repository should be cloned first.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git clone git@github.com:NSO-developer/nso-docker.git Cloning into 'nso-docker'... remote: Enumerating objects: 4597, done. remote: Counting objects: 100% (1579/1579), done. remote: Compressing objects: 100% (635/635), done. remote: Total 4597 (delta 1043), reused 1395 (delta 871), pack-reused 3018 Receiving objects: 100% (4597/4597), 1.36 MiB | 2.08 MiB/s, done. Resolving deltas: 100% (2932/2932), done. $ $ cd nso-docker/ nso-docker$ </code></pre></div></div> <h3 id="download-installation-files">Download Installation Files</h3> <p>To install NSO, you need the installation file, which is not included in the NSO in Docker project. You can download the installation file from the Cisco DevNet page. The installation file is an executable script, which verifies a signature and creates the installer file when executed.</p> <p>You should put the file into the <code class="language-plaintext highlighter-rouge">./nso-install-files</code> directory, which is a default path, where the build script will take a look for the NSO installer.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-docker$ cd nso-install-files nso-install-files$ ls nso-5.5.linux.x86_64.signed.bin </code></pre></div></div> <p>The script should be executed to extract the installer file.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-install-files$ sh nso-5.5.linux.x86_64.signed.bin Unpacking... Verifying signature... Retrieving CA certificate from http://www.cisco.com/security/pki/certs/crcam2.cer ... Successfully retrieved and verified crcam2.cer. Retrieving SubCA certificate from http://www.cisco.com/security/pki/certs/innerspace.cer ... Successfully retrieved and verified innerspace.cer. Successfully verified root, subca and end-entity certificate chain. Successfully fetched a public key from tailf.cer. Successfully verified the signature of nso-5.5.linux.x86_64.installer.bin using tailf.cer </code></pre></div></div> <p>This produces multiple files. All these files should be removed and you should only keep the actual installer in the directory.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-install-files$ ls nso-5.5.linux.x86_64.installer.bin </code></pre></div></div> <h3 id="build-the-base-image">Build the Base Image</h3> <p>Now that the installer is in the <code class="language-plaintext highlighter-rouge">./nso-install-files</code> you are ready to build the base image. You need to run the <code class="language-plaintext highlighter-rouge">make</code> command in the root of the project.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-install-files$ cd .. nso-docker$ make ... OMITTED FOR BREVITY ... </code></pre></div></div> <p>This creates a base image. It is as simple as that.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-docker$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE cisco-nso-base 5.5-foobar 856ffdeef133 6 days ago 564MB cisco-nso-dev 5.5-foobar 04f1f5da57b3 6 days ago 1.4GB </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">make</code> command creates two images:</p> <ul> <li>a production image, which contains only required packages</li> <li>a development image, which also contains documentation, compilers, and other tools</li> </ul> <p>The <code class="language-plaintext highlighter-rouge">make</code> command adds the username to the image tag (<code class="language-plaintext highlighter-rouge">5.5-foobar</code>), but you should add another tag that only includes the NSO version.</p> <h3 id="add-a-new-tag-to-the-base-image">Add a New Tag to the Base Image</h3> <p>The <code class="language-plaintext highlighter-rouge">Makefile</code> comes with another directive (<code class="language-plaintext highlighter-rouge">tag-release</code>), which will be used to tag the image.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-docker$ make NSO_VERSION=5.5 tag-release docker tag cisco-nso-dev:5.5-foobar cisco-nso-dev:5.5 docker tag cisco-nso-base:5.5-foobar cisco-nso-base:5.5 nso-docker$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE cisco-nso-base 5.5 856ffdeef133 6 days ago 564MB cisco-nso-base 5.5-foobar 856ffdeef133 6 days ago 564MB cisco-nso-dev 5.5 04f1f5da57b3 6 days ago 1.4GB cisco-nso-dev 5.5-foobar 04f1f5da57b3 6 days ago 1.4GB </code></pre></div></div> <p>The base image is now ready.</p> <p><br /></p> <h2 id="extend-the-base-image">Extend the Base Image</h2> <p>The base image will be extended with additional packages such as NEDs. The <code class="language-plaintext highlighter-rouge">cisco-nso-base</code> image will be used as a base image. All instructions to extend the base image will be added into the <code class="language-plaintext highlighter-rouge">Dockerfile</code>.</p> <h3 id="create-dockerfile">Create Dockerfile</h3> <p>Let’s first create <code class="language-plaintext highlighter-rouge">Dockerfile</code>. The following Docker image is pretty simple, but it can be more complex than that. The file contains the instructions to extend the <code class="language-plaintext highlighter-rouge">cisco-nso-dev:5.5</code> base image. In the build process, Docker will create a directory and copy NED files from local <code class="language-plaintext highlighter-rouge">neds</code> directory to <code class="language-plaintext highlighter-rouge">/var/opt/ncs/packages/</code> inside the container.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-docker$ cd ../nso-dev-image nso-dev-image$ cat Dockerfile FROM cisco-nso-dev:5.5 RUN mkdir -p /var/opt/ncs/packages/ COPY neds/*.tar.gz /var/opt/ncs/packages/ </code></pre></div></div> <h3 id="download-neds">Download NEDs</h3> <p>You can download NEDs from DevNet and add these as compressed <code class="language-plaintext highlighter-rouge">.tar.gz</code> files to the <code class="language-plaintext highlighter-rouge">neds</code> directory.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ ls neds cisco-ios-cli-6.69.tar.gz cisco-nx-cli-5.21.tar.gz </code></pre></div></div> <h3 id="build-docker-image">Build Docker Image</h3> <p>Now all components are prepared to build an NSO Docker image.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ docker build -t nso:5.5-neds . [+] Building 2.2s (8/8) FINISHED =&gt; [internal] load build definition from Dockerfile 0.0s =&gt; =&gt; transferring dockerfile: 144B 0.0s =&gt; [internal] load .dockerignore 0.0s =&gt; =&gt; transferring context: 2B 0.0s =&gt; [internal] load metadata for docker.io/library/cisco-nso-dev:5.5 0.0s =&gt; [1/3] FROM docker.io/library/cisco-nso-dev:5.5 0.1s =&gt; [internal] load build context 1.4s =&gt; =&gt; transferring context: 83.84MB 1.4s =&gt; [2/3] RUN mkdir -p /var/opt/ncs/packages/ 0.7s =&gt; [3/3] COPY neds/*.tar.gz /var/opt/ncs/packages/ 0.2s =&gt; exporting to image 0.4s =&gt; =&gt; exporting layers 0.4s =&gt; =&gt; writing image sha256:2ee95dcc017a77e3725f984de62c9f896f5c73d8fc00130fd43ca6c0fc2aee6e 0.0s =&gt; =&gt; naming to docker.io/library/nso:5.5-neds 0.0s </code></pre></div></div> <p>After a couple of moments the image is ready. The name of the image is <code class="language-plaintext highlighter-rouge">nso</code> and it is tagged as <code class="language-plaintext highlighter-rouge">5.5-neds</code>.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE nso 5.5-neds 2ee95dcc017a 30 seconds ago 1.48GB cisco-nso-base 5.5 856ffdeef133 6 days ago 564MB cisco-nso-base 5.5-foobar 856ffdeef133 6 days ago 564MB cisco-nso-dev 5.5 04f1f5da57b3 6 days ago 1.4GB cisco-nso-dev 5.5-foobar 04f1f5da57b3 6 days ago 1.4GB </code></pre></div></div> <h3 id="push-the-image-to-the-registry">Push the Image to the Registry</h3> <p>In order to allow other developers to use the same image, the image can be pushed to a registry, such as Docker Hub. To do that, another tag must be added to the image.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ docker tag nso:5.5-neds networktocode/nso:5.5-neds nso-dev-image$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE nso 5.5-neds 2ee95dcc017a 3 minutes ago 1.48GB networktocode/nso 5.5-neds 2ee95dcc017a 3 minutes ago 1.48GB cisco-nso-base 5.5 856ffdeef133 6 days ago 564MB cisco-nso-base 5.5-foobar 856ffdeef133 6 days ago 564MB cisco-nso-dev 5.5 04f1f5da57b3 6 days ago 1.4GB cisco-nso-dev 5.5-foobar 04f1f5da57b3 6 days ago 1.4GB </code></pre></div></div> <p>Once the image is tagged, it can be pushed to the registry.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ docker push networktocode/nso:5.5-neds </code></pre></div></div> <p>From this point, the image is available to anyone that has access to the registry.</p> <p><br /></p> <h2 id="run-a-docker-container">Run a Docker Container</h2> <p>To make everything easier and repeatable, you can create a <code class="language-plaintext highlighter-rouge">Makefile</code>, which contains commands to start and purge NSO containers.</p> <h3 id="create-makefile">Create Makefile</h3> <p>Let’s first create a <code class="language-plaintext highlighter-rouge">Makefile</code>. There are two directives we’ll add to the <code class="language-plaintext highlighter-rouge">Makefile</code>. The first one starts a container, while the other is used to stop and remove the container.</p> <p>There are a couple of options in the <code class="language-plaintext highlighter-rouge">run</code> directive. The one that should be pointed out is the <code class="language-plaintext highlighter-rouge">-v</code> option. This option maps a directory on a local filesystem to a directory in a container. You can map an existing git repository with NSO services to the directory inside the container. NSO loads services from that directory. Because files are originally located on a local filesystem, you can use preferred tools for development, while using Docker as a runtime environment. In this example the <code class="language-plaintext highlighter-rouge">services</code> directory is mapped.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ cat Makefile .PHONY: run run: docker run -itd --name nso \ -v ${PWD}/../services:/nso/run/packages \ -e ADMIN_PASSWORD=admin \ -p 2024:22 \ networktocode/nso:5.5-neds .PHONY: purge purge: docker stop nso docker rm nso </code></pre></div></div> <h3 id="start-a-container">Start a Container</h3> <p>You can now start a container using the <code class="language-plaintext highlighter-rouge">make run</code> command.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ make run </code></pre></div></div> <p>And voila, the container with NSO is running. After a couple of moments the container is ready to use.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3623e63ab2a6 networktocode/nso:5.5-neds "/run-nso.sh" 4 minutes ago Up 4 minutes (healthy) 80/tcp, 443/tcp, 830/tcp, 4334/tcp, 0.0.0.0:2024-&gt;22/tcp nso </code></pre></div></div> <h3 id="connect-to-nso">Connect to NSO</h3> <p>You can now use <code class="language-plaintext highlighter-rouge">ssh</code> to connect to the NSO instance. The port <code class="language-plaintext highlighter-rouge">22</code> in the container is mapped to the port <code class="language-plaintext highlighter-rouge">2024</code> on the host. Let’s open an ssh session to port <code class="language-plaintext highlighter-rouge">2024</code>.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ ssh admin@localhost -p 2024 admin@localhost's password: User admin last logged in 2021-04-26T21:31:21.009543+00:00, to 3623e63ab2a6, from 172.17.0.1 using cli-ssh admin connected from 172.17.0.1 using ssh on 3623e63ab2a6 admin@ncs&gt; </code></pre></div></div> <h3 id="verify-packages">Verify Packages</h3> <p>You can now work with NSO as you would with any other instance running natively on the system or running on a dedicated server. The following example displays the packages that are deployed in NSO.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>admin@ncs&gt; switch cli admin@ncs# show packages package oper-status PACKAGE PROGRAM META FILE CODE JAVA PYTHON BAD NCS PACKAGE PACKAGE CIRCULAR DATA LOAD ERROR NAME UP ERROR UNINITIALIZED UNINITIALIZED VERSION NAME VERSION DEPENDENCY ERROR ERROR INFO ----------------------------------------------------------------------------------------------------------------------------- cisco-ios-cli-6.69 X - - - - - - - - - - cisco-nx-cli-5.21 X - - - - - - - - - - l3vpn X - - - - - - - - - - </code></pre></div></div> <p>There are two NEDs that were added during the build process. Another package called <code class="language-plaintext highlighter-rouge">l3vpn</code> is included in my <code class="language-plaintext highlighter-rouge">services</code> directory, which is mapped to the container. You can make changes to the <code class="language-plaintext highlighter-rouge">l3vpn</code> packages locally on the host, and when you are ready, you can reload packages in NSO and test the new functionality.</p> <h3 id="cleanup">Cleanup</h3> <p>It is also quite simple to remove the development environment. Since you have the <code class="language-plaintext highlighter-rouge">purge</code> directive in the <code class="language-plaintext highlighter-rouge">Makefile</code>, you can simply run the <code class="language-plaintext highlighter-rouge">make purge</code> command to stop and remove the container.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nso-dev-image$ make purge docker stop nso nso docker rm nso nso </code></pre></div></div> <p><br /></p> <h2 id="conclusion">Conclusion</h2> <p>Using Docker for NSO development brings many advantages. It is easy to set up an environment. Developers only need to run a Docker container, while an image is automatically pulled from a Docker registry. The same image can be used later in test and development environments, and you can be pretty sure that all packages and dependencies are installed. You can avoid “it works on my laptop” issues, because the same runtime environment can be used in development and in production.</p> <p>The build process requires more steps, but it can be easily automated. Typically, the process will be running as a job in a CI/CD pipeline.</p> <p>Hopefully this post shows you how to build an NSO development environment using Docker containers.</p> <p>-Uros</p> <p><br /></p> <h2 id="references">References</h2> <ul> <li><a href="https://github.com/NSO-developer/nso-docker">NSO in Docker</a></li> <li><a href="https://developer.cisco.com/docs/nso/#!getting-and-installing-nso">Getting NSO</a></li> </ul>Uros BajzeljDeveloping services for Cisco NSO requires a development environment, which can be set up locally on your laptop or on a dedicated server. When a developer starts working on an NSO service, it has to install a correct NSO version, all required NEDs, and dependencies. This is where Docker can come in handy. It is a tool to make packaging easy, which means that you can pre-build a Docker image, which contains NSO, NEDs, and all dependencies.Setting Up Nautobot ChatOps with Microsoft Teams2021-04-22T00:00:00+00:002021-04-22T00:00:00+00:00https://blog.networktocode.com/post/setting-up-nautobot-chatops-with-msteams<p>Network to Code has released a number of amazing plugins for Nautobot. One of which, adding ChatOps functionality, can be found <a href="https://github.com/nautobot/nautobot-plugin-chatops">here</a> on GitHub. This plugin adds ChatOps capabilities directly into your existing ChatOps client, in the form of a chat bot, and supports four of the more popular services available right now. The four services currently supported are Slack, Microsoft Teams, Webex Teams, and Mattermost.</p> <p>If this is your first time hearing about ChatOps or this plugin, you can see the <a href="https://www.youtube.com/watch?v=_AfHe05Y3DA">ChatOps demo</a> on YouTube or join <a href="slack.networktocode.com">slack.networktocode.com</a> and try it out for yourself in the <code class="language-plaintext highlighter-rouge">#nautobot-chat</code> channel.</p> <p>Today, I’ll be going over how to get this plugin working in Nautobot and how to get a chat bot up and running for Microsoft Teams. The process is fairly different from the other three providers listed, and slightly more complex, but the end results are amazing. Let’s dive right in!</p> <p> </p> <h2 id="getting-started">Getting Started</h2> <p>There are two main parts to getting the ChatOps plugin working with any ChatOps service: configuring it on the ChatOps service directly, and installing and configuring it on your Nautobot server. Microsoft Teams splits the first part into two sections: creating the service in Azure, and installing the app in the Teams client.</p> <p>For simplicity, I will already assume you have the base Nautobot server installed and working. If not, you can find the full documentation over on <a href="https://nautobot.readthedocs.io/en/latest/">readthedocs.io</a>, or join our public Slack channel <code class="language-plaintext highlighter-rouge">#nautobot</code> at <a href="slack.networktocode.com">slack.networktocode.com</a> and ask for assistance.</p> <p> </p> <h2 id="part-1-configuring-microsoft-teams-saas">Part 1: Configuring Microsoft Teams SaaS</h2> <h3 id="azure-and-permissions">Azure and Permissions</h3> <p>To start off, I will be configuring a brand-new bot for Microsoft Teams from scratch. Microsoft runs their bots differently from Slack, Webex Teams, or Mattermost, in that their bot service runs on Azure. If you don’t have a Microsoft Azure account, you will need to create one or get access to it through your company before continuing.</p> <p>According to the Microsoft <a href="https://docs.microsoft.com/en-us/azure/bot-service/bot-service-resources-faq-security?view=azure-bot-service-4.0">docs</a>, you will need “Contributor access either in the subscription or in a specific resource group. A user with the Contributor role in a resource group can create a new bot in that specific resource group. A user in the Contributor role for a subscription can create a bot in a new or existing resource group.”</p> <h3 id="configuring-azure">Configuring Azure</h3> <p>There are three main parts to configuring a bot in Azure:</p> <p>1) Create a Bot Service and Resource Group</p> <p>2) Configure the Bot Service channel</p> <p>3) Create a Client Secret for the Bot Service</p> <p>I’ll break down each part individually, with step-by-step instructions and screenshots along the way.</p> <h3 id="1---create-a-bot-service-and-resource-group">1 - Create a Bot Service and Resource Group</h3> <p>First, log into the Azure Portal at <a href="https://portal.azure.com">https://portal.azure.com</a>.</p> <p>At the top of the screen is a search bar. Search for “Bot Channels Registration”, then select the option with the same name under “Marketplace” on the right side. This will take you to the page to create a new Bot Service.</p> <blockquote> <p><em>NOTE: You may need to activate this service first within your company’s Azure subscription, which is not covered in this post.</em></p> </blockquote> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/01_azure_search_bot_channel_registration.png" alt="Azure - Search Bot Channels Registration" /></p> <p>A few key fields to fill out when creating a new Bot Service are:</p> <ul> <li><strong>Bot Handle</strong> - What you want your bot handle to be. This is <em>not</em> what your bot is called in the MS Teams client, or how users will interact with your bot, but it is unique (case-insensitive) within the overall Azure Bot Framework.</li> <li><strong>Resource Group</strong> - If there’s an existing one you want to use, select it. Otherwise, select the “Create new” link and create a new resource group. In this example, I’m creating a new Resource Group called “RG_nautobot_ntcblog”.</li> <li><strong>Pricing Tier</strong> - I’m using F0, which is Azure’s basic free tier.</li> <li><strong>Messaging Endpoint</strong> - Enter your Nautobot service URL in this format: <code class="language-plaintext highlighter-rouge">https://&lt;server&gt;/api/plugins/chatops/ms_teams/messages/</code>.</li> </ul> <p>In this demo example, I’m using the <a href="https://ngrok.com/">Ngrok</a> service. For a production Nautobot server, you would enter in the publicly facing DNS endpoint for inbound webhooks into your Nautobot server.</p> <ul> <li><strong>Subscription, Location, and Application Insights</strong> - Use whatever is appropriate for your company and situation.</li> </ul> <p>Once this is all filled out, select the <strong>Create</strong> button at the bottom of the screen.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/02_azure_create_bot_service.png" alt="Azure - Create Bot Service" /></p> <p>Wait until the deployment is done and successful before continuing. You can monitor its progress in the upper right of the Azure dashboard, under the alerts icon (looks like a bell).</p> <h3 id="2---configure-the-bot-service-channel">2 - Configure the Bot Service Channel</h3> <p>Back on the Azure Portal homepage, in the main search bar, search for “Bot Services”, then click on the corresponding link under the Services section on the left side.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/03_azure_search_bot_services.png" alt="Azure - Search Bot Services" /></p> <p>Select the name of the bot handle you just created.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/04_azure_select_bot_handle.png" alt="Azure - Select Bot Handle" /></p> <p>Select the Microsoft Teams client icon, as circled in the screenshot below.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/05_azure_select_teams_channel.png" alt="Azure - Select Teams Channel" /></p> <p>All of the options on the next <code class="language-plaintext highlighter-rouge">Configure Microsoft Teams</code> page should be ok when left to default, but should be reviewed anyway for your specific use case.</p> <p>Once done, click Save at the bottom of the page, and review and Agree to any ToS popups.</p> <h3 id="3---create-a-client-secret-for-the-bot-service">3 - Create a Client Secret for the Bot Service</h3> <p>Next, on the left sidebar, select Configuration under the Settings section. Save the <code class="language-plaintext highlighter-rouge">Microsoft App ID</code> somewhere else, as you’ll need this later.</p> <p>Select the “Manage” link directly above the App ID.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/06_azure_configure_bot.png" alt="Azure - Select Teams Channel" /></p> <p>This will take you to the Certificates &amp; Secrets page.</p> <p>Click <code class="language-plaintext highlighter-rouge">New client secret</code>. Name it something descriptive, configure the expiration setting as necessary, and click Add.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/07_azure_client_secret.png" alt="Azure - Select Teams Channel" /></p> <p>Once it’s created, it will appear in the Client Secrets table at the bottom of the page. Copy and save the newly generated secret for later, as there’s no way to recover it once you leave the page.</p> <blockquote> <p><em>NOTE: If you lose the key or copy it incorrectly, you will have to return to this page and generate a new secret.</em></p> </blockquote> <h3 id="azure-recap">Azure Recap</h3> <p>At this point, the Nautobot ChatOps plugin is fully set up within Azure. You should have two pieces of information for later use: the App ID and the Client Secret.</p> <p> </p> <h2 id="part-2-installing-and-configuring-the-nautobot-chatops-plugin">Part 2: Installing and Configuring the Nautobot ChatOps Plugin</h2> <p>Next, you must install and configure the Nautobot ChatOps plugin on your Nautobot server. Luckily, the fine folks at Network to Code have made this process incredibly simple!</p> <h3 id="installing-the-plugin">Installing the Plugin</h3> <p>First, log into your Nautobot server and change to the user account Nautobot is running as. From there, it’s as simple as installing the plugin via a <code class="language-plaintext highlighter-rouge">pip install</code> command.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo</span> <span class="nt">-iu</span> nautobot <span class="nv">$ </span>pip3 <span class="nb">install </span>nautobot-chatops </code></pre></div></div> <p>Once the package is installed, the plugin will need to be enabled in your <code class="language-plaintext highlighter-rouge">nautobot_config.py</code>. If Nautobot was originally set up according to the default installation docs, this file will be located at <code class="language-plaintext highlighter-rouge">/opt/nautobot/nautobot_config.py</code>. In this file, add in the name of the plugins to the <strong>PLUGINS</strong> variable, then configure the required settings in the <strong>PLUGINS_CONFIG</strong> variable below it.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">PLUGINS</span> <span class="o">=</span> <span class="p">[</span><span class="s">"nautobot_chatops"</span><span class="p">]</span> <span class="n">PLUGINS_CONFIG</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"nautobot_chatops"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"enable_ms_teams"</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">"microsoft_app_id"</span><span class="p">:</span> <span class="s">"&lt;app_id&gt;"</span><span class="p">,</span> <span class="s">"microsoft_app_password"</span><span class="p">:</span> <span class="s">"&lt;client_secret&gt;"</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>Make sure to replace <code class="language-plaintext highlighter-rouge">&lt;app_id&gt;</code> and <code class="language-plaintext highlighter-rouge">&lt;client_secret&gt;</code> with the App ID and Client Secret saved from Azure in the previous steps. Then save the file and restart the Nginx and Nautobot services.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>systemctl restart nginx <span class="nb">sudo </span>systemctl restart nautobot-worker.service </code></pre></div></div> <h3 id="configuring-the-plugin-in-nautobot">Configuring the Plugin in Nautobot</h3> <p>Next, we need to configure the plugin in Nautobot to accept commands. For most deployments, open and unrestricted access to the bot from any chat account is undesirable. Therefore, access to the chatbot defaults to “deny all” when initially installed. Permissions for individual organizations, channels, and users must be set up here. For the purposes of this blog post, we will grant all access.</p> <p>First, log into your Nautobot server. If this is the first plugin installed, a new menu option will appear at the top called Plugins. Under it, in section Nautobot ChatOps, select Access Grants.</p> <p>Select the Add button to create a new access grant.</p> <ul> <li>Command - You can specify permissions on a command-by-command basis, or specify all commands with an asterisk <code class="language-plaintext highlighter-rouge">*</code> as a wildcard. Example commands: <code class="language-plaintext highlighter-rouge">nautobot</code> or <code class="language-plaintext highlighter-rouge">clear</code></li> <li>Subcommand - You can specify permissions for subcommands as well, or all subcommands with an asterisk <code class="language-plaintext highlighter-rouge">*</code>. Example subcommands: <code class="language-plaintext highlighter-rouge">get-devices</code> or <code class="language-plaintext highlighter-rouge">help</code></li> <li>Grant Type - You need to create permissions for all three options: Organization, Channel, and User. <ul> <li>Organization - This is for permissions specific to your organization. This is good for configuring allowed/blocked commands organization-wide.</li> <li>Channel - This is for configuring permissions on a per-channel basis.</li> <li>User - This is for configuring permissions on a per-user basis.</li> </ul> </li> <li>Name - The corresponding name of the organization, channel, or user. This is used more like a description, whereas the value below is used when interacting with the MS Teams SaaS API on the backend.</li> <li>Value - Corresponding ID value to grant access to. Enter an asterisk <code class="language-plaintext highlighter-rouge">*</code> to grant access to all organizations, channels, or users.</li> </ul> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/08_nautobot_new_access_grant.png" alt="Nautobot - New Access Grant" /></p> <p>Once all three permissions are created, the plugin is done being set up in Nautobot. The minimum amount of permissions required are three. You can allow everyone in your organization access to all commands (not recommended) by using wildcards for organization, channel, and user permissions.</p> <p>In the above example, here’s how I’ve set it up:</p> <ul> <li>Organization - The org <em>only</em> has access to the <code class="language-plaintext highlighter-rouge">nautobot</code> command. It does not have access to <code class="language-plaintext highlighter-rouge">clear</code>, or any future commands the plugin may end up supporting.</li> <li>User - Anyone can run just the <code class="language-plaintext highlighter-rouge">nautobot get-devices</code> command, however user John Doe can run any command. Note that he cannot run <code class="language-plaintext highlighter-rouge">clear</code>, as that is restricted at the Organization permission above.</li> <li>Channel - Anyone can accss the bot from any channel, but again, only the <code class="language-plaintext highlighter-rouge">nautobot get-devices</code> command. However, anyone in channel <code class="language-plaintext highlighter-rouge">bot-admins</code> can access any command available to them.</li> </ul> <p>To summarize, anyone can run <code class="language-plaintext highlighter-rouge">nautobot get-devices</code>, whereas John Doe and anyone in the channel Bot Admins can run any <code class="language-plaintext highlighter-rouge">nautobot</code> subcommand. Nobody can run <code class="language-plaintext highlighter-rouge">clear</code> or any command that doesn’t start with <code class="language-plaintext highlighter-rouge">nautobot</code>.</p> <p>The last step is configuring the Microsoft Teams client.</p> <p> </p> <h2 id="part-3-installing-and-configuring-the-app-in-teams">Part 3: Installing and Configuring the App in Teams</h2> <p>The last main step needed is uploading and installing the app into your Microsoft Teams client for use within your organization.</p> <p>Before continuing, you need to download a single ZIP file from the ChatOps plugin repo, found <a href="https://github.com/nautobot/nautobot-plugin-chatops/blob/develop/Nautobot_ms_teams.zip">here</a>. This will be used later for ease of installing the app into the client.</p> <p>The ZIP file contains three files:</p> <p>1) manifest.json - Pre-configured information for the bot</p> <p>2) color.png - Image to use for the bot</p> <p>3) outline.png - Transparent image to use for the bot</p> <p>Open up your Microsoft Teams desktop client and log in if you haven’t already.</p> <p>On the left sidebar and select Apps. Use the Apps search bar to search for “App Studio”, then select the tile and click Open to open the App Studio. If it isn’t currently installed, install it first, then open it.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/09_teams_open_app_studio.png" alt="Teams - Open App Studio" /></p> <p>If not already selected, select Manifest Editor on the top horizontal menu bar, then select Import an existing app. Upload the <code class="language-plaintext highlighter-rouge">Nautobot_ms_teams.zip</code> file that you downloaded earlier.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/10_teams_import_existing_app.png" alt="Teams - Import An Existing App" /></p> <p>Once imported, the Edit an app page will appear, allowing you to configure the settings for the bot.</p> <p><strong>Required Setting Changes</strong></p> <p>The App ID setting must be updated in two different locations. This ZIP file comes pre-loaded with an example App ID, but it must be replaced with the one for your specific bot, as created and saved from Azure in the previous steps. All other settings can be left as is, but feel free to review them as desired.</p> <p>1) Under section 1 Details, select App details. Update the App ID to the value that you saved from Azure earlier.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/11_teams_configure_app_id_1.png" alt="Teams - Configure App ID Location 1" /></p> <p>2) Under section 2 Capabilities, select page Bots. Select the Edit button next to the Bot at the top. In the pop-out window, in the field Connect to a different bot id, update the App ID to the value that you saved from Azure earlier. Then click Save.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/12_teams_configure_app_id_2.png" alt="Teams - Configure App ID Location 2" /></p> <p>3) Under section 3 Finish, select page Test and distribute. Select the Download button to download the app as a .zip file to your computer.</p> <p>NOTE: You can attempt to Install the app instead of downloading it and re-uploading it in step 4, however it requires permissions to do so, which I had trouble with even when I was the administrator of the Microsoft Teams environment (e.g. in my free-tier personal environment).</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/13_teams_test_distribute_app.png" alt="Teams - Download App" /></p> <p>4) The app, with the now-updated App ID’s, will download to the <code class="language-plaintext highlighter-rouge">~/Downloads</code> folder (or equivalent) on your computer. The file should be named Nautobot.zip (or, if you changed the name of the bot in the manifest, the name you gave it).</p> <p>Once downloaded, select Apps on the bottom of the left-hand sidebar, then scroll to the bottom and select Upload a Custom App. If your company requires approval of custom apps, it will be submitted for approval before being installed.</p> <p><img src="../../../static/images/blog_posts/nautobot_chatops_msteams_setup/14_teams_upload_custom_app.png" alt="Teams - Upload Custom App" /></p> <p>5) The app package will then upload into MS Teams as an install app, and you’ll be taken to the Apps page with the bot (app) listed under Built by your org. Select the bot/app tile, then click the blue Add button.</p> <p> </p> <h2 id="done">Done!</h2> <p>That’s it! Your new Nautobot ChatOps plugin should now be installed for your Microsoft Teams client, and usable by anyone with the appropriate permissions (configured earlier in part 2).</p> <p>You can do some really cool things with the bot once it’s up and running, and you have some data in Nautobot. You can send the message <code class="language-plaintext highlighter-rouge">nautobot help</code> to the app (no <code class="language-plaintext highlighter-rouge">/</code> forward slash) to see a list of all supported commmands.</p> <h2 id="interacting-with-nautobot-in-microsoft-teams">Interacting with Nautobot in Microsoft Teams</h2> <p>There are currently a couple of ways to interact with the Nautobot plugin by default directly in the Microsoft Teams client, although these can be modified in the app permissions in the same area where you installed the app originally (in part 3). They are:</p> <p>1) Chat - In the main left sidebar, select Chat, then search for “Nautobot” (or whatever you renamed the bot to). You can message the bot directly here.</p> <p>2) App - In the main left sidebar, select the three dots, then in the pop-out menu, search for “Nautobot” and select it. I recommend right-clicking the icon in the left sidebar once the window opens to pin it for future interactions.</p> <p> </p> <h2 id="forward-looking">Forward Looking</h2> <p>Here at Network to Code, as we continue developing Nautobot, we will be adding functionality to this ChatOps plugin as well. With the code publicly available <a href="https://github.com/nautobot/nautobot-plugin-chatops">here</a> on GitHub, we encourage anyone looking to contribute to do so and join our growing open-source community around Nautobot!</p> <p>Additionally, there’s a blog post from earlier this month around creating your own custom chat commands within this plugin. If interested, you can read it <a href="https://blog.networktocode.com/post/creating-custom-chat-commands-using-nautobot-chatops/">here</a>.</p> <p>Thanks for reading, and I hope you enjoy ChatOps as much as I do!</p> <p>-Matt</p>Matt VitaleNetwork to Code has released a number of amazing plugins for Nautobot. One of which, adding ChatOps functionality, can be found here on GitHub. This plugin adds ChatOps capabilities directly into your existing ChatOps client, in the form of a chat bot, and supports four of the more popular services available right now. The four services currently supported are Slack, Microsoft Teams, Webex Teams, and Mattermost.Ansible - BGP Networking Troubleshooting Guide2021-04-20T00:00:00+00:002021-04-20T00:00:00+00:00https://blog.networktocode.com/post/ansible-bgp-network-troubleshooting-guide<p>Network troubleshooting is a common automation use case. Network outages are costly and time-consuming and often require the network engineers to log into network equipment and manually investigate network issues. Working on network operations teams, I quickly noticed that troubleshooting network problems is a playbook of repeatable steps, hence the rationale for automating network troubleshooting with Ansible.</p> <h2 id="use-case---bgp">Use Case - BGP</h2> <p>Troubleshooting Layer 3 connectivity tends to lead an operations engineer to jump into multiple routers and check routing. Let’s say internet access has been lost from the WAN edge. If I were troubleshooting this, my instincts would tell me to go to my edge router(s) and check the BGP neighbor going towards my ISP.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>east-rtr#show ip bgp summary &lt;...output omitted...&gt; Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 4.4.4.4 4 400 0 0 0 0 0 08:11 Idle </code></pre></div></div> <p>From the output of <code class="language-plaintext highlighter-rouge">show ip bgp summary</code> the issue can determined, <strong>BGP</strong> is down toward the ISP. How can Ansible help? This is a simplified example with one router and one WAN connection, but what happens if you have 10, 15, or more BGP relationships you need to check. It is costly to manually log in to each router to check the status of BGP. How can Ansible help?</p> <h3 id="checking-bgp-with-ansible">Checking BGP with Ansible</h3> <p>Here is a sequential listing of what the Ansible playbook is doing.</p> <ol> <li>Run <code class="language-plaintext highlighter-rouge">show ip bgp summary</code> outputs from ISP routers.</li> <li>Use <code class="language-plaintext highlighter-rouge">ansible-napalm</code> to get BGP facts on the neighbors for easy reporting.</li> <li>Create an easy-to-consume report using a Jinja2 template to create a report with BGP neighbor status.</li> <li>Assemble all the device reports into a single overview report.</li> <li>Iterate through the neighbors and <strong>if</strong> a neighbor is down, attempt to ping the destination IP to verify Layer 3 reachability using <code class="language-plaintext highlighter-rouge">napalm-ping</code>.</li> </ol> <h3 id="pre-req">Pre-req</h3> <p>There needs to be a valid Ansible inventory, either a static inventory file or dynamic inventories utilizing an existing SoT (Source of Truth). For demonstration purposes a static file will be used.</p> <p><code class="language-plaintext highlighter-rouge">inventory.cfg</code></p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[isp_routers]</span> <span class="nn">[isp_routers:vars]</span> <span class="py">ansible_network_os</span><span class="p">=</span><span class="s">ios</span> <span class="nn">[isp_routers:children]</span> <span class="err">east_isp</span> <span class="err">west_isp</span> <span class="nn">[east_isp]</span> <span class="err">east-rtr</span> <span class="nn">[west_isp]</span> <span class="err">west-rtr</span> </code></pre></div></div> <blockquote> <p>For help building an inventory file. See <a href="https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html">Ansible Inventory</a></p> </blockquote> <h3 id="step-1">Step 1</h3> <p>Create a simple playbook to execute <code class="language-plaintext highlighter-rouge">show ip bgp neighbors</code> on all of the routers in the group called <code class="language-plaintext highlighter-rouge">isp_routers</code>.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">SUMMARY"</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">False</span> <span class="na">connection</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_cli"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">isp_routers"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'SHOW</span><span class="nv"> </span><span class="s">IP</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">SUMMARY'"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">ip</span><span class="nv"> </span><span class="s">bgp</span><span class="nv"> </span><span class="s">summary"</span> <span class="na">register</span><span class="pi">:</span> <span class="s2">"</span><span class="s">output_ios"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">output_ios.stdout[0]</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>Running the playbook results in the following output.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>▶ ansible-playbook pb.yml -u ntc -k SSH password: PLAY [PLAY:1 - GET BGP SUMMARY] ************************************************************************************************************************************************************************************** TASK [TASK:1 - 'SHOW IP BGP SUMMARY'] ******************************************************************************************************************************************************************************** ok: [east-rtr] ok: [west-rtr] TASK [TASK:2 - PRINT BGP OUTPUT] ************************************************************************************************************************************************************************************* ok: [east-rtr] =&gt; { "msg": "BGP router identifier 1.1.1.1, local AS number 100\nBGP table version is 416, main routing table version 416\n28 network entries using 6944 bytes of memory\n41 path entries using 5576 bytes of memory\n8/7 BGP path/bestpath attribute entries using 2304 bytes of memory\n4 BGP AS-PATH entries using 128 bytes of memory\n0 BGP route-map cache entries using 0 bytes of memory\n0 BGP filter-list cache entries using 0 bytes of memory\nBGP using 14952 total bytes of memory\nBGP activity 124/96 prefixes, 232/191 paths, scan interval 60 secs\n32 networks peaked at 23:40:21 Jan 7 2021 UTC (6w5d ago)\n\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd\n4.4.4.4 4 400 0 0 0 0 0 08:21 Idle" } ok: [west-rtr] =&gt; { "msg": "BGP router identifier 2.2.2.2, local AS number 100\nBGP table version is 579, main routing table version 579\n28 network entries using 6944 bytes of memory\n41 path entries using 5576 bytes of memory\n8/7 BGP path/bestpath attribute entries using 2304 bytes of memory\n4 BGP AS-PATH entries using 128 bytes of memory\n0 BGP route-map cache entries using 0 bytes of memory\n0 BGP filter-list cache entries using 0 bytes of memory\nBGP using 14952 total bytes of memory\nBGP activity 158/130 prefixes, 267/226 paths, scan interval 60 secs\n32 networks peaked at 23:40:21 Jan 7 2021 UTC (6w5d ago)\n\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd\n8.8.8.8 4 400 0 0 0 0 0 18:52 1" } PLAY RECAP *********************************************************************************************************************************************************************************************************** east-rtr : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 west-rtr : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre></div></div> <p>At this point you have a single pane to quickly check all the BGP neighbors; however, it’s hard to read the output. To take this playbook to the next level, we can easily take command output and create structured data using one of the various cli parsing modules.</p> <ul> <li><a href="https://docs.ansible.com/ansible/latest/network/user_guide/cli_parsing.html">Ansible Parsing</a></li> <li>Parsing Strategies on NTC Blog by Mikhail Yohman. <ul> <li><a href="http://blog.networktocode.com/post/parsing-strategies-intro/">Parsing Strategies - An Introduction</a></li> </ul> </li> </ul> <p>What if more information is needed? You could check <code class="language-plaintext highlighter-rouge">route</code> counts or <code class="language-plaintext highlighter-rouge">layer 3 reachability</code>.</p> <p>Let’s dig into this use case further.</p> <h3 id="step-2">Step 2</h3> <p>Use <code class="language-plaintext highlighter-rouge">napalm-ansible</code> module to run <code class="language-plaintext highlighter-rouge">get-facts</code> on BGP.</p> <p>Note: For readability the rest of the task will be in a <code class="language-plaintext highlighter-rouge">PLAY:2</code> of the playbook.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">USE</span><span class="nv"> </span><span class="s">NAPALM</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">FACTS"</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">False</span> <span class="na">connection</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_cli"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">isp_routers"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'GET</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">FACTS'"</span> <span class="na">napalm_get_facts</span><span class="pi">:</span> <span class="na">filter</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">bgp_neighbors"</span> <span class="na">register</span><span class="pi">:</span> <span class="s2">"</span><span class="s">bgp"</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="s">var=bgp</span> </code></pre></div></div> <p>Results in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>▶ ansible-playbook napalm_pb.yml -u ntc -k SSH password: PLAY [PLAY:2 - USE NAPALM BGP FACTS] ********************************************************************************************************************************************************************************* TASK [TASK:1 - 'GET BGP FACTS'] ************************************************************************************************************************************************************************************** ok: [east-rtr] ok: [west-rtr] TASK [debug] ********************************************************************************************************************************************************************************************************* ok: [east-rtr]] =&gt; { "bgp": { "ansible_facts": { "napalm_bgp_neighbors": { "global": { "peers": { "4.4.4.4": { "address_family": { "ipv4 unicast": { "accepted_prefixes": 0, "received_prefixes": 0, "sent_prefixes": 0 } }, "description": "", "is_enabled": true, "is_up": false, "local_as": 100, "remote_as": 400, "remote_id": "4.4.4.4", "uptime": 0 }, "router_id": "1.1.1.1" } } }, "changed": false, "failed": false } } ok: [west-rtr] =&gt; { "bgp": { "ansible_facts": { "napalm_bgp_neighbors": { "global": { "peers": { "8.8.8.8": { "address_family": { "ipv4 unicast": { "accepted_prefixes": 1, "received_prefixes": 1, "sent_prefixes": 11 } }, "description": "", "is_enabled": true, "is_up": true, "local_as": 100, "remote_as": 400, "remote_id": "8.8.8.8", "uptime": 1641600 }, "router_id": "2.2.2.2" } } }, "changed": false, "failed": false } } PLAY RECAP *********************************************************************************************************************************************************************************************************** east-rtr : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 west-rtr : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 </code></pre></div></div> <p>The playbook returns structured operational data on the BGP neighbors. This data can easily be used to build a report.</p> <h3 id="step-3">Step 3</h3> <p>Create a report with the validation of BGP.</p> <p>We can take the registered data from <code class="language-plaintext highlighter-rouge">TASK:1</code> and pass it to the <code class="language-plaintext highlighter-rouge">template</code> module where a Jinja2 template can be used to create a report.</p> <p>We will add <code class="language-plaintext highlighter-rouge">TASK:2</code> to our PLAY.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'GENERATE</span><span class="nv"> </span><span class="s">REPORTS'"</span> <span class="na">template</span><span class="pi">:</span> <span class="na">src</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./templates/bgp_report.j2"</span> <span class="na">dest</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./build/{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}.txt"</span> </code></pre></div></div> <p>An example of a Jinja2 template can be seen below:</p> <p><code class="language-plaintext highlighter-rouge">bgp_report.j2</code></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">Hostname</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">inventory_hostname</span> <span class="pi">}}</span> <span class="s">-----------------</span> <span class="pi">{</span><span class="err">%</span> <span class="nv">for neighbor</span><span class="pi">,</span> <span class="nv">details in bgp</span><span class="pi">[</span><span class="s2">"</span><span class="s">ansible_facts"</span><span class="pi">][</span><span class="s2">"</span><span class="s">napalm_bgp_neighbors"</span><span class="pi">][</span><span class="s2">"</span><span class="s">global"</span><span class="pi">][</span><span class="s2">"</span><span class="s">peers"</span><span class="pi">]</span><span class="nv">.items() %</span><span class="pi">}</span> <span class="na">Neighbor</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">neighbor</span> <span class="pi">}}</span> <span class="na">Enabled</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">details</span><span class="pi">[</span><span class="s2">"</span><span class="s">is_enabled"</span><span class="pi">]</span> <span class="pi">}}</span> <span class="na">Neighbor_UP</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">details</span><span class="pi">[</span><span class="s2">"</span><span class="s">is_up"</span><span class="pi">]</span> <span class="pi">}}</span> <span class="na">Accepted Prefixes</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">details</span><span class="pi">[</span><span class="s1">'</span><span class="s">address_family'</span><span class="pi">][</span><span class="s1">'</span><span class="s">ipv4</span><span class="nv"> </span><span class="s">unicast'</span><span class="pi">][</span><span class="s1">'</span><span class="s">accepted_prefixes'</span><span class="pi">]</span> <span class="pi">}}</span> <span class="na">Received Prefixes</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">details</span><span class="pi">[</span><span class="s1">'</span><span class="s">address_family'</span><span class="pi">][</span><span class="s1">'</span><span class="s">ipv4</span><span class="nv"> </span><span class="s">unicast'</span><span class="pi">][</span><span class="s1">'</span><span class="s">received_prefixes'</span><span class="pi">]</span> <span class="pi">}}</span> <span class="na">Sent Prefixes</span><span class="pi">:</span> <span class="pi">{{</span> <span class="nv">details</span><span class="pi">[</span><span class="s1">'</span><span class="s">address_family'</span><span class="pi">][</span><span class="s1">'</span><span class="s">ipv4</span><span class="nv"> </span><span class="s">unicast'</span><span class="pi">][</span><span class="s1">'</span><span class="s">sent_prefixes'</span><span class="pi">]</span> <span class="pi">}}</span> <span class="pi">{</span><span class="err">%</span> <span class="nv">endfor %</span><span class="pi">}</span> </code></pre></div></div> <p>The Jinja2 template will render the structured data into a human-readable format.</p> <p>Example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Hostname: east-rtr ----------------- Neighbor: 4.4.4.4 Enabled: True Neighbor_UP: False Accepted Prefixes: 0 Received Prefixes: 0 Sent Prefixes: 0 </code></pre></div></div> <h3 id="step-4">Step 4</h3> <p>With multiple devices in our inventory group, a file per device will be written. Parsing through multiple files can slow down the time to resolution; therefore, merging all these files together into one all-encompassing report will be done in the next task.</p> <p>The Ansible <code class="language-plaintext highlighter-rouge">assemble</code> module will be used to merge all the reports together.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:3</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">ASSEMBLE</span><span class="nv"> </span><span class="s">REPORTING</span><span class="nv"> </span><span class="s">FROM</span><span class="nv"> </span><span class="s">HOST</span><span class="nv"> </span><span class="s">DETAILS"</span> <span class="na">assemble</span><span class="pi">:</span> <span class="na">src</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./build"</span> <span class="c1"># Directory with files to merge.</span> <span class="na">dest</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./reports/report.txt"</span> <span class="c1"># Merged output filename.</span> </code></pre></div></div> <p>Once <code class="language-plaintext highlighter-rouge">TASK:3</code> executes, one report is generated with the following output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Hostname: east-rtr ----------------- Neighbor: 4.4.4.4 Enabled: True Neighbor_UP: False Accepted Prefixes: 0 Received Prefixes: 0 Sent Prefixes: 0 Hostname: west-rtr ----------------- Neighbor: 8.8.8.8 Enabled: True Neighbor_UP: True Accepted Prefixes: 8 Received Prefixes: 8 Sent Prefixes: 22 </code></pre></div></div> <p>Now a single easy-to-read file exists to look at neighbors. We see <code class="language-plaintext highlighter-rouge">east-rtr</code> has a BGP neighbor that is <code class="language-plaintext highlighter-rouge">DOWN</code>.</p> <h3 id="step-5">Step 5</h3> <p>Check whether any <code class="language-plaintext highlighter-rouge">DOWN</code> neighbor is reachable via ping.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:4</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PING</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">NEIGHBORS</span><span class="nv"> </span><span class="s">THAT</span><span class="nv"> </span><span class="s">ARE</span><span class="nv"> </span><span class="s">DOWN"</span> <span class="na">napalm_ping</span><span class="pi">:</span> <span class="na">hostname</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_user</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_password</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">dev_os</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_network_os</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">destination</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item.key</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">bgp['ansible_facts']['napalm_bgp_neighbors']['global']['peers']</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">dict2items</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">when</span><span class="pi">:</span> <span class="s2">"</span><span class="s">not</span><span class="nv"> </span><span class="s">item.value.is_up"</span> <span class="na">register</span><span class="pi">:</span> <span class="s">neighbor_down</span> </code></pre></div></div> <p>After the reachability check is completed, print the results for the <code class="language-plaintext highlighter-rouge">DOWN</code> neighbors.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:5</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">PING</span><span class="nv"> </span><span class="s">RESULTS</span><span class="nv"> </span><span class="s">FOR</span><span class="nv"> </span><span class="s">DOWN</span><span class="nv"> </span><span class="s">NEIGHBORS"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['ping_results']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">neighbor_down['results']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">when</span><span class="pi">:</span> <span class="s2">"</span><span class="s">item['ping_results']</span><span class="nv"> </span><span class="s">is</span><span class="nv"> </span><span class="s">defined"</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">TASK:5</code> example output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "msg": { "success": { "packet_loss": 5, "probes_sent": 5, "results": [], "rtt_avg": 0.0, "rtt_max": 0.0, "rtt_min": 0.0, "rtt_stddev": 0.0 } } } </code></pre></div></div> <h3 id="playbook-summary">Playbook Summary</h3> <p>Valuable troubleshooting data was gathered by running this playbook. A BGP neighbor is down on <code class="language-plaintext highlighter-rouge">east-rtr</code>. Details about all neighbors were also collected, including: enabled state, current neighbor state, and sent/received route counts. Finally, for any <code class="language-plaintext highlighter-rouge">DOWN</code> neighbors a reachability check using ping was performed. Most importantly, all this data was assembled across all our <code class="language-plaintext highlighter-rouge">isp_routers</code> in just seconds. This was still a simplified example with only two routers, but extrapolating this across tens, hundreds, or more routers is very powerful.</p> <p>It is important to mention that additional tasks could be added to this playbook to troubleshoot further, for example:</p> <ul> <li>Check the routing to the neighbor IP.</li> <li>Grab the next-hop IP from the route entry.</li> <li>Verify that the ARP table for the next-hop IP has a MAC entry.</li> </ul> <h2 id="full-playbook">Full Playbook</h2> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">SUMMARY"</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">False</span> <span class="na">connection</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_cli"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">isp_routers"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'SHOW</span><span class="nv"> </span><span class="s">IP</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">SUMMARY'"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">ip</span><span class="nv"> </span><span class="s">bgp</span><span class="nv"> </span><span class="s">summary"</span> <span class="na">register</span><span class="pi">:</span> <span class="s2">"</span><span class="s">output_ios"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">output_ios.stdout[0]</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">USE</span><span class="nv"> </span><span class="s">NAPALM</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">FACTS"</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">False</span> <span class="na">connection</span><span class="pi">:</span> <span class="s2">"</span><span class="s">network_cli"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s2">"</span><span class="s">isp_routers"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:1</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'GET</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">FACTS'"</span> <span class="na">napalm_get_facts</span><span class="pi">:</span> <span class="s">filter="bgp_neighbors"</span> <span class="na">register</span><span class="pi">:</span> <span class="s2">"</span><span class="s">bgp"</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="s">var=bgp</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:2</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">'GENERATE</span><span class="nv"> </span><span class="s">REPORT'"</span> <span class="na">template</span><span class="pi">:</span> <span class="na">src</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./templates/bgp_report.j2"</span> <span class="na">dest</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./build/{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}.txt"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:3</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">ASSEMBLE</span><span class="nv"> </span><span class="s">REPORTING</span><span class="nv"> </span><span class="s">FROM</span><span class="nv"> </span><span class="s">HOST</span><span class="nv"> </span><span class="s">DETAILS"</span> <span class="na">assemble</span><span class="pi">:</span> <span class="na">src</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./build"</span> <span class="na">dest</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./reports/report.txt"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:4</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PING</span><span class="nv"> </span><span class="s">BGP</span><span class="nv"> </span><span class="s">NEIGHBORS</span><span class="nv"> </span><span class="s">THAT</span><span class="nv"> </span><span class="s">ARE</span><span class="nv"> </span><span class="s">DOWN"</span> <span class="na">napalm_ping</span><span class="pi">:</span> <span class="na">hostname</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_user</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_password</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">dev_os</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_network_os</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">destination</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['key']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">with_dict</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">bgp['ansible_facts']['napalm_bgp_neighbors']['global']['peers']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">when</span><span class="pi">:</span> <span class="s2">"</span><span class="s">not</span><span class="nv"> </span><span class="s">item['value']['is_up']"</span> <span class="na">register</span><span class="pi">:</span> <span class="s2">"</span><span class="s">neighbor_down"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK:5</span><span class="nv"> </span><span class="s">-</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">PING</span><span class="nv"> </span><span class="s">RESULTS</span><span class="nv"> </span><span class="s">FOR</span><span class="nv"> </span><span class="s">DOWN</span><span class="nv"> </span><span class="s">NEIGHBORS"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['ping_results']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">neighbor_down['results']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">when</span><span class="pi">:</span> <span class="s2">"</span><span class="s">item['ping_results']</span><span class="nv"> </span><span class="s">is</span><span class="nv"> </span><span class="s">defined"</span> </code></pre></div></div> <h2 id="summary">Summary</h2> <p>BGP troubleshooting is one of a multitude of operational troubleshooting playbooks that could be executed for troubleshooting connectivity issues. Taking these same steps to other use cases can greatly improve <code class="language-plaintext highlighter-rouge">MTTR</code> on network issues and outages. Furthermore, these playbooks can be extended using a module to update ITSM ticket notes, or even for use during an existing daily <code class="language-plaintext highlighter-rouge">network readiness</code> task.</p> <p>-Jeff</p>Jeff KalaNetwork troubleshooting is a common automation use case. Network outages are costly and time-consuming and often require the network engineers to log into network equipment and manually investigate network issues. Working on network operations teams, I quickly noticed that troubleshooting network problems is a playbook of repeatable steps, hence the rationale for automating network troubleshooting with Ansible.Programmatic Nautobot GraphQL Requests via Pynautobot2021-04-15T00:00:00+00:002021-04-15T00:00:00+00:00https://blog.networktocode.com/post/programmatic-nautobot-graphql-requests-via-pynautobot<p>Thanks to its ability to efficiently allow the request of specific information that can span multiple resources, GraphQL is a powerful query tool. A single GraphQL API request can return information that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.</p> <p>A <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">prior blog post in this series</a> covered how to craft Nautobot GraphQL queries using the Nautobot GraphiQL interface. Recall that the GraphiQL interface is just for testing the GraphQL queries. This post will build on that knowledge, showing you how to leverage those queries to craft remote GraphQL requests for programmatic use via the open source <code class="language-plaintext highlighter-rouge">pynautobot</code> Python library. The <code class="language-plaintext highlighter-rouge">pynautobot</code> project page is <a href="https://pypi.org/project/pynautobot/">here</a>; the GitHub repository can be found <a href="https://github.com/nautobot/pynautobot">here</a>.</p> <p>The <code class="language-plaintext highlighter-rouge">pynautobot</code> Python library is an API wrapper that allows easy interactions with <em>Nautobot</em>. This specific article will focus the GraphQL capabilities within the library.</p> <p>The examples in this post all use the public Nautobot demo site https://demo.nautobot.com/. Readers are encouraged to follow along.</p> <h2 id="authentication">Authentication</h2> <p>Security tokens are typically required for programmatic access to Nautobot’s data. The following sections cover how to obtain a token and how to leverage it within Postman to craft GraphQL API calls.</p> <h3 id="tokens">Tokens</h3> <p>Nautobot security tokens are created in the Web UI. To view your token(s), navigate to the API Tokens page under your user profile. If there is no token present, create one or get the necessary permissions to do so.</p> <p><img src="../../../static/images/blog_posts/graphQL/graphQL-pynautobot/1-api-token.png" alt="" /></p> <h2 id="getting-started-with-pynautobot">Getting Started with Pynautobot</h2> <h3 id="installing-pynautobot">Installing Pynautobot</h3> <p><code class="language-plaintext highlighter-rouge">pynautobot</code> is a Python library. From your environment install it via <code class="language-plaintext highlighter-rouge">pip3</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% pip3 install pynautobot Collecting pynautobot Downloading pynautobot-1.0.1-py3-none-any.whl (29 kB) . . . Successfully installed pynautobot-1.0.1 % </code></pre></div></div> <h3 id="using-pynautbot">Using Pynautbot</h3> <p>This first example will walk through a <code class="language-plaintext highlighter-rouge">pynautobot</code> GraphQL query in a Python interpreter.</p> <p>Start a Python shell and import <code class="language-plaintext highlighter-rouge">pynautobot</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% python3 Python 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. &gt;&gt;&gt; &gt;&gt;&gt; &gt;&gt;&gt; import pynautobot &gt;&gt;&gt; </code></pre></div></div> <p>Taking a quick look at Classes within <code class="language-plaintext highlighter-rouge">pynautobot</code>, <code class="language-plaintext highlighter-rouge">api</code> is the one we will want to use. The <code class="language-plaintext highlighter-rouge">help</code> tells us that the <code class="language-plaintext highlighter-rouge">api</code> Class can take <code class="language-plaintext highlighter-rouge">url</code> and <code class="language-plaintext highlighter-rouge">token</code> as parameters.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; dir(pynautobot) ['AllocationError', 'ContentError', 'DistributionNotFound', 'RequestError', &lt;&lt; dunders snipped &gt;&gt;, 'api', 'core', 'get_distribution', 'models'] &gt;&gt;&gt; &gt;&gt;&gt; help(pynautobot.api) Help on class Api in module pynautobot.core.api: class Api(builtins.object) | Api(url, token=None, threading=False) . . . </code></pre></div></div> <blockquote> <p>NOTE: the token from the https://demo.nautobot.com server is <code class="language-plaintext highlighter-rouge">aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</code></p> </blockquote> <p>Define a value <code class="language-plaintext highlighter-rouge">token</code> with the token value from your Nautobot implementation and a value <code class="language-plaintext highlighter-rouge">url</code> for your Nautobot server.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; url = "https://demo.nautobot.com" &gt;&gt;&gt; &gt;&gt;&gt; token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" &gt;&gt;&gt; </code></pre></div></div> <p>This live example will use a query from a previous post. Define <code class="language-plaintext highlighter-rouge">query</code> in Python using the exact text from the query used in example 3 in the <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">GraphiQL post</a>:</p> <blockquote> <p>NOTE: the query text can be found in the last section of the blog, under <strong>Text Query Examples and Results</strong></p> </blockquote> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; query = """ ... query { ... devices(site:"ams") { ... name ... } ... } ... """ &gt;&gt;&gt; &gt;&gt;&gt; query '\nquery {\n devices(site:"ams") {\n name\n }\n}\n' &gt;&gt;&gt; &gt;&gt;&gt; print(query) query { devices(site:"ams") { name } } &gt;&gt;&gt; </code></pre></div></div> <p>Now construct the <code class="language-plaintext highlighter-rouge">pynautobot.api</code> Class with the parameters we have:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; &gt;&gt;&gt; nb = pynautobot.api(url, token) &gt;&gt;&gt; </code></pre></div></div> <p>Notice that the <code class="language-plaintext highlighter-rouge">nb</code> Object instance has a <code class="language-plaintext highlighter-rouge">graphql</code> attribute . . .</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; dir(nb) ['__class__', &lt;&lt; dunders snipped &gt;&gt;, 'base_url', 'circuits', 'dcim', 'extras', 'graphql', 'headers', 'http_session', 'ipam', 'openapi', 'plugins', 'status', 'tenancy', 'threading', 'token', 'users', 'version', 'virtualization'] &gt;&gt;&gt; </code></pre></div></div> <p>. . . and <code class="language-plaintext highlighter-rouge">graphql</code> has a <code class="language-plaintext highlighter-rouge">query</code> method:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; dir(nb.graphql) ['__class__', &lt;&lt; dunders snipped &gt;&gt;, 'api', 'query', 'url'] &gt;&gt;&gt; &gt;&gt;&gt; help(nb.graphql.query) Help on method query in module pynautobot.core.graphql: query(query: str, variables: Optional[Dict[str, Any]] = None) -&gt; pynautobot.core.graphql.GraphQLRecord method of pynautobot.core.graphql.GraphQLQuery instance Runs query against Nautobot Graphql endpoint. Args: query (str): Query string to send to the API variables (dict): Dictionary of variables to use with the query string, defaults to None Raises: GraphQLException: - When the query string is invalid. TypeError: - When `query` passed in is not of type string. - When `variables` passed in is not a dictionary. Exception: - When unknown error is passed in, please open an issue so this can be addressed. Examples: &gt;&gt;&gt; try: ... response.raise_for_status() ... except Exception as e: ... variable = e ... &gt;&gt;&gt; variable &gt;&gt;&gt; variable.response.json() {'errors': [{'message': 'Cannot query field "nae" on type "DeviceType". Did you mean "name" or "face"?', 'locations': [{'line': 4, 'column': 5}]}]} &gt;&gt;&gt; variable.response.status_code 400 Returns: GraphQLRecord: Response of the API call </code></pre></div></div> <p>Using the structure above, create the query and explore the results.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; &gt;&gt;&gt; response = nb.graphql.query(query=query) &gt;&gt;&gt; &gt;&gt;&gt; dir(response) ['__class__', &lt;&lt; dunders snipped &gt;&gt;, 'json', 'status_code'] &gt;&gt;&gt; &gt;&gt;&gt; response.status_code 200 &gt;&gt;&gt; &gt;&gt;&gt; response.json {'data': {'devices': [{'name': 'ams-edge-01'}, {'name': 'ams-edge-02'}, {'name': 'ams-leaf-01'}, {'name': 'ams-leaf-02'}, {'name': 'ams-leaf-03'}, {'name': 'ams-leaf-04'}, {'name': 'ams-leaf-05'}, {'name': 'ams-leaf-06'}, {'name': 'ams-leaf-07'}, {'name': 'ams-leaf-08'}]}} &gt;&gt;&gt; </code></pre></div></div> <p>Here is <code class="language-plaintext highlighter-rouge">response</code> in a more readable format:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; from pprint import pprint &gt;&gt;&gt; pprint(response.json) {'data': {'devices': [{'name': 'ams-edge-01'}, {'name': 'ams-edge-02'}, {'name': 'ams-leaf-01'}, {'name': 'ams-leaf-02'}, {'name': 'ams-leaf-03'}, {'name': 'ams-leaf-04'}, {'name': 'ams-leaf-05'}, {'name': 'ams-leaf-06'}, {'name': 'ams-leaf-07'}, {'name': 'ams-leaf-08'}]}} &gt;&gt;&gt; </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">response</code> object can be parsed, making the data accessible to your Python code:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; &gt;&gt;&gt; devices = response.json['data']['devices'] &gt;&gt;&gt; &gt;&gt;&gt; devices[2] {'name': 'ams-leaf-01'} &gt;&gt;&gt; </code></pre></div></div> <h2 id="a-full-pynautobot-script">A Full Pynautobot Script</h2> <p>This next example features a full script, using an example from the <a href="https://blog.networktocode.com/post/graphql-aliasing-with-nautobot/">GraphQL Aliasing with Nautobot</a> post in this series. You can also see the YouTube video that accompanies the post <a href="https://youtu.be/3T4cRsIqM7Y">here</a>.</p> <p>The script below features a query that uses GraphQL <em>aliasing</em> to request device names for multiple sites in a single query. The device names in each site will be returned grouped by the site name; each device name will be alaised with an <code class="language-plaintext highlighter-rouge">inventory_hostname</code> key (instead of <code class="language-plaintext highlighter-rouge">name</code>) for use in an Ansible environment.</p> <blockquote> <p>The query text for this script was pulled directly from the <strong>Aliasing Solves the Problem</strong> section of the referenced blog post and copied directly into the <code class="language-plaintext highlighter-rouge">query</code> variable.</p> </blockquote> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import json import pynautobot from pprint import pprint print("Querying Nautobot via pynautobot.") print() url = "https://demo.nautobot.com" print("url is: {}".format(url)) print() query = """ query { ams_devices:devices(site:"ams") { inventory_hostname:name } sin_devices:devices(site:"sin") { inventory_hostname:name } bkk_devices:devices(site:"bkk") { inventory_hostname:name } } """ print("query is:") print(query) print() token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" nb = pynautobot.api(url, token) response = nb.graphql.query(query=query) response_data = response.json print("Here is the response data in json:") pprint(response_data) print() print("Here are the bkk devices:") pprint(response_data['data']['bkk_devices']) </code></pre></div></div> <p>Here is the script’s output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% python3 -i graphql_ams_query_pynautobot.py Querying Nautobot via pynautobot. url is: https://demo.nautobot.com query is: query { ams_devices:devices(site:"ams") { inventory_hostname:name } sin_devices:devices(site:"sin") { inventory_hostname:name } bkk_devices:devices(site:"bkk") { inventory_hostname:name } } Here is the response data in json: {'data': {'ams_devices': [{'inventory_hostname': 'ams-edge-01'}, {'inventory_hostname': 'ams-edge-02'}, {'inventory_hostname': 'ams-leaf-01'}, {'inventory_hostname': 'ams-leaf-02'}, {'inventory_hostname': 'ams-leaf-03'}, {'inventory_hostname': 'ams-leaf-04'}, {'inventory_hostname': 'ams-leaf-05'}, {'inventory_hostname': 'ams-leaf-06'}, {'inventory_hostname': 'ams-leaf-07'}, {'inventory_hostname': 'ams-leaf-08'}], 'bkk_devices': [{'inventory_hostname': 'bkk-edge-01'}, {'inventory_hostname': 'bkk-edge-02'}, {'inventory_hostname': 'bkk-leaf-01'}, {'inventory_hostname': 'bkk-leaf-02'}, {'inventory_hostname': 'bkk-leaf-03'}, {'inventory_hostname': 'bkk-leaf-04'}, {'inventory_hostname': 'bkk-leaf-05'}, {'inventory_hostname': 'bkk-leaf-06'}, {'inventory_hostname': 'bkk-leaf-07'}, {'inventory_hostname': 'bkk-leaf-08'}], 'sin_devices': [{'inventory_hostname': 'sin-edge-01'}, {'inventory_hostname': 'sin-edge-02'}, {'inventory_hostname': 'sin-leaf-01'}, {'inventory_hostname': 'sin-leaf-02'}, {'inventory_hostname': 'sin-leaf-03'}, {'inventory_hostname': 'sin-leaf-04'}, {'inventory_hostname': 'sin-leaf-05'}, {'inventory_hostname': 'sin-leaf-06'}, {'inventory_hostname': 'sin-leaf-07'}, {'inventory_hostname': 'sin-leaf-08'}]}} Here are the bkk devices: [{'inventory_hostname': 'bkk-edge-01'}, {'inventory_hostname': 'bkk-edge-02'}, {'inventory_hostname': 'bkk-leaf-01'}, {'inventory_hostname': 'bkk-leaf-02'}, {'inventory_hostname': 'bkk-leaf-03'}, {'inventory_hostname': 'bkk-leaf-04'}, {'inventory_hostname': 'bkk-leaf-05'}, {'inventory_hostname': 'bkk-leaf-06'}, {'inventory_hostname': 'bkk-leaf-07'}, {'inventory_hostname': 'bkk-leaf-08'}] &gt;&gt;&gt; </code></pre></div></div> <h2 id="wrapping-up">Wrapping Up</h2> <p>To fully leverage GraphQL’s efficiency, it must be used programmatically. The <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">first post</a> in this series demonstrated using Nautobot’s <em>GraphiQL</em> interface to craft GraphQL queries. This post builds on that by showing how to convert those GraphQL queries into remote requests in Python code for programmatic use.</p> <p>To find out more about <code class="language-plaintext highlighter-rouge">pynautobot</code>, including the additional capabilities not described in this post, start with the <a href="https://github.com/nautobot/pynautobot">pynautobot Github repo</a>.</p>Tim FiolaThanks to its ability to efficiently allow the request of specific information that can span multiple resources, GraphQL is a powerful query tool. A single GraphQL API request can return information that would otherwise require literally dozens of individual REST queries and extensive data filtering, post-processing, and isolation.Deploying Services with Docker, NGINX, Route 53 &amp; Let’s Encrypt2021-04-13T00:00:00+00:002021-04-13T00:00:00+00:00https://blog.networktocode.com/post/hosting-services-with-docker-and-nginx<p>Docker is a power tool for deploying applications or services, and there are numerous Docker orchestration tools available that can help to simplify the management of the deployed containers. But what if you are wanting to deploy a small number of services and not wanting to undertake setting up and managing another application stack just to run a handful of containers. I will cover how I deployed a handful of services on a single Docker host. The services I deployed are LetsEncrypt to generate a wildcard certificate, Route 53 to register <strong>A</strong> and <strong>CNAME</strong> records, and NGINX to do reverse proxy with SNI encapsulation. I previously had some of these services deployed in containers on a Raspberry Pi as part of my <a href="/post/InfluxDB/">Aquarium Controller</a>, but I wanted to provide better flexibility for deployment and not pigeonhole myself to deploying only ARM-compatible containers. That, combined with wanting persistent services deployed at home, is what led me to building a new physical Linux host running Ubtunu 20.04 LTS &amp; Docker.</p> <p>This post is meant to show the flexibility of Docker not a production guide on deploying with Docker. I fully recommend using orchestration when deploying containers and would have deployed via container orchestration if I were working with multiple hosts or was managing more services.</p> <p>In this post, I will be using my InfluxDB service as the example service. This service is used from both outside the Docker host via NGINX reverse proxy and for east/west container-to-container communication. Also, all of my services are defined as code via <code class="language-plaintext highlighter-rouge">docker-compose</code> to provide an easier experience vs raw <code class="language-plaintext highlighter-rouge">docker run</code> commands.</p> <h2 id="communication">Communication</h2> <p>Below is an example of a communication flow for a user consuming a service deployed in Docker that then consumes another backend service on the same host. The end user is none the wiser on how the traffic is flowing, and as the administrator you are able to take advantage of container-to-container communication.</p> <p><img src="../../../static/images/blog_posts/docker-nginx-letsencrypt/grafana-communication.png" alt="" /></p> <h3 id="route-53">Route 53</h3> <p>I am using Route 53 to register all of my DNS records; this simplifies the amount of services I am managing on premise by not running a service like <code class="language-plaintext highlighter-rouge">Bind 9 DNS</code>. All of the records I am publishing in Route53 resolve to private IP addresses and are not routeable from outside my network. In my examples I have a single A record for the physical host itself and all services are <strong>CNAME</strong> records pointing to the server’s <strong>A</strong> record. My domain name was registered with Route 53, which helps to streamline the process.</p> <h3 id="docker-container-name-resolution">Docker Container Name Resolution</h3> <p>Docker provides networks that are internal to the Docker daemon and the ability to perform container name resolution for containers that are on the same Docker network. To simplify the declaration of these supporting services, I am using <code class="language-plaintext highlighter-rouge">docker-compose</code>; and to communicate east/west within containers I only have to send traffic to the adjacent service name. In the above diagram, the Grafana service communicates to the InfluxDB service via <code class="language-plaintext highlighter-rouge">http://influxdb:8086/</code>. Docker resolves the hostname <code class="language-plaintext highlighter-rouge">influxdb</code> to the IP address of the InfluxDB container.</p> <h3 id="lets-encrypt">Let’s Encrypt</h3> <p>Although the services are deployed on my local home network and are behind an appropriate firewall, I prefer to deploy services with TLS from day one. This is a best practice that was instilled in me from day one of my days in enterprise environments. In my example, I am using CertBot to request and manage my certificate. To simplify certificate management I am using a wildcard cert of <code class="language-plaintext highlighter-rouge">*.whitej6.com</code> for all services. Also, an extension of TLS is <code class="language-plaintext highlighter-rouge">server name indication</code> or <code class="language-plaintext highlighter-rouge">SNI</code></p> <h3 id="nginx-and-sni">NGINX and SNI</h3> <p>All the services that are meant to be consumed from outside the Docker daemon are only exposed to localhost within the port definition of the service. This ensures ALL traffic that I want to allow into Docker must come from the physical host or an application that is deployed to the host. Each of the services that I need to expose have their own definition in an NGINX configuration file. The configuration tells NGINX which certificate to use, which requested server name maps to which underlying localhost port number. When NGINX receives an HTTPS request it first determines which service is requested via SNI and then performs a reverse proxy to the correct port on localhost. This allows me to terminate TLS on the physical host and run plain text protocols from NGINX to the underlying Docker service. By performing my own TLS termination in a secure manner outside containers, it can simplify container deployment, reduce the need to customize a vendor-provided container, and/or figure out how each vendor performs TLS termination in their containers.</p> <h2 id="example-deployment">Example Deployment</h2> <p>The example used is an actual depiction of services I am hosting within my home.</p> <h3 id="prerequisites">Prerequisites</h3> <p>This example will not be covering how to install Ubuntu, Docker, docker-compose, CertBot, or NGINX. These items have well documented installations and should be referenced. The example will cover the consumption of these services.</p> <h3 id="route-53-1">Route 53</h3> <p>I am using Route 53 to host DNS records needed for the applications deployed in the Docker stack. Below is a table of the two records that are needed for deploying. At a later step in the example, I will create a third record that is required for generating my certificate.</p> <table> <thead> <tr> <th>Record Type</th> <th>Source</th> <th>Destination</th> </tr> </thead> <tbody> <tr> <td>A</td> <td>ubuntu-server.whitej6.com</td> <td>10.0.0.16</td> </tr> <tr> <td>CNAME</td> <td>influxdb.whitej6.com</td> <td>ubuntu-server.whitej6.com</td> </tr> </tbody> </table> <p>You can see in the output that <code class="language-plaintext highlighter-rouge">influxdb.whitej6.com</code> is a CNAME for <code class="language-plaintext highlighter-rouge">ubuntu-server.whitej6.com</code> that resolves to <code class="language-plaintext highlighter-rouge">10.0.0.16</code></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ dig influxdb.whitej6.com <span class="p">;</span> &lt;&lt;<span class="o">&gt;&gt;</span> DiG 9.10.6 &lt;&lt;<span class="o">&gt;&gt;</span> influxdb.whitej6.com <span class="p">;;</span> global options: +cmd <span class="p">;;</span> Got answer: <span class="p">;;</span> -&gt;&gt;HEADER<span class="o">&lt;&lt;-</span> <span class="no">opcode</span><span class="sh">: QUERY, status: NOERROR, id: 61162 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;influxdb.whitej6.com. IN A ;; ANSWER SECTION: influxdb.whitej6.com. 236 IN CNAME ubuntu-server.whitej6.com. ubuntu-server.whitej6.com. 238 IN A 10.0.0.16 ;; Query time: 21 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Apr 06 13:44:41 CDT 2021 ;; MSG SIZE rcvd: 93 ➜ ~ </span></code></pre></div></div> <h3 id="generating-certificate">Generating Certificate</h3> <p>Using Cerbot to generate my Let’s Encrypt certificate provides a simple end-to-end solution for generating and managing a signed certificate. There are a few different challenges that can be used to validate domain ownership. Since I have minimal experience with Certbot and I can easily administer my DNS records in my domain, I chose to use a DNS challenge. In the example below you will see I issue the command and Certbot responds with a TXT record I need to create for <code class="language-plaintext highlighter-rouge">_acme-challenge.whitej6.com</code> with a value of <code class="language-plaintext highlighter-rouge">XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code>. After I create the record, I can then hit <code class="language-plaintext highlighter-rouge">Enter</code> to continue and Certbot will perform the challenge and validate whether the value of the TXT record matches what was expected. Let’s Encrypt certificates are short-lived certificates that last 90 days. Certbot supports renewing the certificates via <code class="language-plaintext highlighter-rouge">certbot renew</code> command.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ <span class="nb">sudo </span>certbot <span class="nt">-d</span> <span class="s2">"*.whitej6.com"</span> <span class="nt">--manual</span> <span class="nt">--preferred-challenges</span> dns certonly Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Requesting a certificate <span class="k">for</span> <span class="k">*</span>.whitej6.com Performing the following challenges: dns-01 challenge <span class="k">for </span>whitej6.com - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Please deploy a DNS TXT record under the name _acme-challenge.whitej6.com with the following value: XXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Before continuing, verify the record is deployed. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue Waiting <span class="k">for </span>verification... Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/whitej6.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/whitej6.com/privkey.pem Your certificate will expire on 2021-07-05. To obtain a new or tweaked version of this certificate <span class="k">in </span>the future, simply run certbot again. To non-interactively renew <span class="k">*</span>all<span class="k">*</span> of your certificates, run <span class="s2">"certbot renew"</span> - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let<span class="s1">'s Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le ➜ ~ </span></code></pre></div></div> <h3 id="deploying-containers">Deploying Containers</h3> <p>The deployment and management of containers is done via <code class="language-plaintext highlighter-rouge">docker-compose</code> to simplify the management of Docker.</p> <h4 id="influxdbdocker-composeyml">~/influxdb/docker-compose.yml</h4> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span> <span class="na">services</span><span class="pi">:</span> <span class="na">influxdb</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s">influxdb:latest</span> <span class="na">restart</span><span class="pi">:</span> <span class="s">always</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">127.0.0.1:8086:8086"</span> <span class="na">volumes</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">/influxdb:/var/lib/influxdb"</span> </code></pre></div></div> <p>I want to ensure each deployed service can communicate with other services as needed. To accomplish this, I am using the same compose project name in the deployments. If each service is deployed in another compose project, the name resolution and east/west communication becomes complicated. In the output below, you will see Docker is not too terribly happy to see services declared outside the docker-compose.yml file, this is to be expected.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ <span class="nb">export </span><span class="nv">COMPOSE_PROJECT_NAME</span><span class="o">=</span><span class="s2">"server"</span> ➜ ~ docker-compose <span class="nt">-f</span> influxdb/docker-compose.yml up <span class="nt">-d</span> Building with native build. Learn about native build <span class="k">in </span>Compose here: https://docs.docker.com/go/compose-native-build/ WARNING: Found orphan containers <span class="o">(</span>server_grafana_1, server_gitlab_1, server_redis_1, server_registry_1, server_netbox_1, server_postgres_1<span class="o">)</span> <span class="k">for </span>this project. If you removed or renamed this service <span class="k">in </span>your compose file, you can run this <span class="nb">command </span>with the <span class="nt">--remove-orphans</span> flag to clean it up. Starting server_influxdb_1 ... <span class="k">done</span> ➜ ~ </code></pre></div></div> <h3 id="configuring-nginx">Configuring NGINX</h3> <p>I personally like keeping my base NGINX configuration file default and overload what I need with individual <code class="language-plaintext highlighter-rouge">.conf</code> files in the <code class="language-plaintext highlighter-rouge">conf.d</code> folder. Each override I declare in it’s own file to make it easier for me to identify and make changes. The file paths below are based on the default install location of NGINX. Your install may use different paths, but the concept is the same.</p> <h4 id="etcnginxconfdredirectconf">/etc/nginx/conf.d/redirect.conf</h4> <p>Forces all HTTP traffic to redirect to HTTPS and keeps the requested host intact.</p> <div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">server</span> <span class="p">{</span> <span class="kn">listen</span> <span class="mi">80</span> <span class="s">default_server</span><span class="p">;</span> <span class="kn">return</span> <span class="mi">301</span> <span class="s">https://</span><span class="nv">$host$request_uri</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <h4 id="etcnginxconfdinfluxdbconf">/etc/nginx/conf.d/influxdb.conf</h4> <div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">map</span> <span class="nv">$http_x_forwarded_proto</span> <span class="nv">$thescheme</span> <span class="p">{</span> <span class="kn">default</span> <span class="nv">$scheme</span><span class="p">;</span> <span class="kn">https</span> <span class="s">https</span><span class="p">;</span> <span class="p">}</span> <span class="k">server</span> <span class="p">{</span> <span class="c1"># Inbound requested hostname</span> <span class="kn">server_name</span> <span class="s">influxdb.whitej6.com</span><span class="p">;</span> <span class="kn">listen</span> <span class="mi">443</span> <span class="s">ssl</span><span class="p">;</span> <span class="c1"># Let's Encrypt certificate location</span> <span class="kn">ssl_certificate</span> <span class="n">/etc/letsencrypt/live/whitej6.com/fullchain.pem</span><span class="p">;</span> <span class="kn">ssl_certificate_key</span> <span class="n">/etc/letsencrypt/live/whitej6.com/privkey.pem</span><span class="p">;</span> <span class="kn">client_max_body_size</span> <span class="mi">25m</span><span class="p">;</span> <span class="kn">proxy_set_header</span> <span class="s">X-Forwarded-Host</span> <span class="nv">$http_host</span><span class="p">;</span> <span class="kn">proxy_set_header</span> <span class="s">X-Real-IP</span> <span class="nv">$remote_addr</span><span class="p">;</span> <span class="kn">proxy_set_header</span> <span class="s">X-Forwarded-Proto</span> <span class="nv">$thescheme</span><span class="p">;</span> <span class="kn">add_header</span> <span class="s">P3P</span> <span class="s">'CP="ALL</span> <span class="s">DSP</span> <span class="s">COR</span> <span class="s">PSAa</span> <span class="s">PSDa</span> <span class="s">OUR</span> <span class="s">NOR</span> <span class="s">ONL</span> <span class="s">UNI</span> <span class="s">COM</span> <span class="s">NAV"'</span><span class="p">;</span> <span class="c1"># Where to reverse proxy HTTP traffic to</span> <span class="kn">location</span> <span class="n">/</span> <span class="p">{</span> <span class="kn">proxy_pass</span> <span class="s">http://localhost:8086</span><span class="p">;</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <h4 id="restart-nginx">Restart NGINX</h4> <p>Once the configuration is complete for NGINX, you need to restart the NGINX service. I am using <code class="language-plaintext highlighter-rouge">systemctl</code> to manage running system services on the host.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ <span class="nb">sudo </span>systemctl restart nginx ➜ ~ </code></pre></div></div> <h3 id="validating-stack">Validating Stack</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ influxdb <span class="nb">echo</span> <span class="s2">"curl from outside the docker network"</span> curl from outside the docker network ➜ influxdb curl https://influxdb.whitej6.com/health <span class="o">{</span><span class="s2">"checks"</span>:[],<span class="s2">"message"</span>:<span class="s2">"ready for queries and writes"</span>,<span class="s2">"name"</span>:<span class="s2">"influxdb"</span>,<span class="s2">"status"</span>:<span class="s2">"pass"</span>,<span class="s2">"version"</span>:<span class="s2">"1.8.4"</span><span class="o">}</span>% ➜ influxdb ➜ influxdb <span class="nb">echo</span> <span class="s2">"curl from inside the docker network"</span> curl from inside the docker network ➜ influxdb docker <span class="nb">exec</span> <span class="nt">-it</span> server_gitlab_1 sh <span class="nt">-c</span> <span class="s2">"curl http://influxdb:8086/health"</span> <span class="o">{</span><span class="s2">"checks"</span>:[],<span class="s2">"message"</span>:<span class="s2">"ready for queries and writes"</span>,<span class="s2">"name"</span>:<span class="s2">"influxdb"</span>,<span class="s2">"status"</span>:<span class="s2">"pass"</span>,<span class="s2">"version"</span>:<span class="s2">"1.8.4"</span>% ➜ influxdb </code></pre></div></div> <p>With all the services deployed and verified I have a fully functioning multi-tenanted Docker host securely serving each service over HTTPS with DNS records to improve useability. This has enabled me to move the Grafana and InfluxDB service off the Raspberry Pi hosting my aquarium controller to the Docker host to improve user experience. Hopefully this post helps you in securely deploying services to a Docker host.</p>Jeremy WhiteDocker is a power tool for deploying applications or services, and there are numerous Docker orchestration tools available that can help to simplify the management of the deployed containers. But what if you are wanting to deploy a small number of services and not wanting to undertake setting up and managing another application stack just to run a handful of containers. I will cover how I deployed a handful of services on a single Docker host. The services I deployed are LetsEncrypt to generate a wildcard certificate, Route 53 to register A and CNAME records, and NGINX to do reverse proxy with SNI encapsulation. I previously had some of these services deployed in containers on a Raspberry Pi as part of my Aquarium Controller, but I wanted to provide better flexibility for deployment and not pigeonhole myself to deploying only ARM-compatible containers. That, combined with wanting persistent services deployed at home, is what led me to building a new physical Linux host running Ubtunu 20.04 LTS &amp; Docker.GraphQL Aliasing with Nautobot2021-04-08T00:00:00+00:002021-04-08T00:00:00+00:00https://blog.networktocode.com/post/graphql-aliasing-with-nautobot<p>GraphQL <em>aliasing</em> is a very potent feature that allows the user to customize the query results by specifying key names in the returned data.</p> <p>GraphQL is very efficient, in large part because it allows a single query to access info from multiple resources and returns only the specific information the user requests. This greatly reduces the number of required queries and eliminates the need for post-query data filtering.</p> <p>GraphQL <em>aliasing</em> is another feature within GraphQL that increases efficiency. This post will demonstrate GraphQL aliasing with two real-world use cases and examples.</p> <p>A <a href="https://blog.networktocode.com/post/leveraging-the-power-of-graphql-with-nautobot/">recent Network to Code blog post</a> covered how to construct GraphQL queries using Nautobot’s Graph<strong>i</strong>QL interface. This post will build on that prior post, so if you have not read that prior post yet and are not familiar with GraphQL, I’d recommend reading that before going further because this article builds on those concepts.</p> <h2 id="what-is-graphql-aliasing">What Is GraphQL <em>Aliasing</em>?</h2> <p>GraphQL <em>aliasing</em> allows the user to customize the names of data keys in the returned data. This is helpful in situations where the user is working in a pre-existing data architecture that expects specific key names or where it’s helpful to have key names that conform to the existing architecture’s naming conventions.</p> <p>This is perhaps better explained with examples, so let’s get to those. The reader can follow along with these examples at the Nautobot public demo sandbox at https://demo.nautobot.com/. Log in with the posted username and password on the site.</p> <h2 id="an-ansible-context">An Ansible Context</h2> <p>Ansible is a powerful abstraction and orchestration tool. Working in that context, it’s helpful to have Nautobot return data that conforms to Ansible’s naming conventions.</p> <h3 id="a-baseline-query">A Baseline Query</h3> <p>Here is a GraphQL query in Nautobot that uses a <em>query parameter</em> and returns all device names in a site:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">ams_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"ams"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <blockquote> <p>NOTE: The <em>query</em> keyword is optional. The above query can also be conducted with <em>query</em> omitted, as such:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="n">ams_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"ams"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div> </div> </blockquote> <p>Here is the returned data for this baseline example:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">data</span><span class="err">":</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">ams_devices</span><span class="err">":</span><span class="w"> </span><span class="err">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-03"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-04"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-05"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-06"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-07"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">name</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-08"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <h3 id="query-with-aliasing">Query with Aliasing</h3> <p>Since we’re in an Ansible context in this example, it’s likely more useful to have the device names come back with an <code class="language-plaintext highlighter-rouge">inventory_hostname</code> key instead of the <code class="language-plaintext highlighter-rouge">name</code> key since <code class="language-plaintext highlighter-rouge">inventory_hostname</code> matches up well with Ansible terminology and use cases.</p> <p>Here is an example that does that. Notice the aliasing, done by prepending <code class="language-plaintext highlighter-rouge">inventory_hostname:</code> to the queried <code class="language-plaintext highlighter-rouge">name</code> parameter:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="n">ams_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"ams"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">inventory_hostname</span><span class="p">:</span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>This is the data that gets returned:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">data</span><span class="err">":</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">ams_devices</span><span class="err">":</span><span class="w"> </span><span class="err">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-03"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-04"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-05"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-06"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-07"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-08"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Pretty slick! Again, there is no need to post-process the data since GraphQL returned only the data we asked for and gave it the specific key name we want.</p> <h2 id="executing-multiple-targeted-queries">Executing Multiple Targeted Queries</h2> <p>Aliasing is also useful for executing what would be multiple targeted queries in a single request.</p> <h3 id="the-problem-statement">The Problem Statement</h3> <p>Building on the example above, imagine that you need <code class="language-plaintext highlighter-rouge">inventory_hostname</code> values from multiple specific sites. You want the <code class="language-plaintext highlighter-rouge">inventory_hostname</code> data grouped by each queried site.</p> <p>Querying for multiple <code class="language-plaintext highlighter-rouge">device</code> sets, each with different parameters, with only a single query produces errors:</p> <p><img src="../../../static/images/blog_posts/graphQL/graphql_alias/bad_query.png" alt="" /></p> <p>GraphQL does not like the multiple <code class="language-plaintext highlighter-rouge">devices</code> queries because each query has a different parameter (<code class="language-plaintext highlighter-rouge">site</code>, in this example).</p> <h3 id="aliasing-solves-the-problem">Aliasing Solves the Problem</h3> <p>Aliasing allows the user to execute those multiple queries in a single request. In this use case, we will alias each <code class="language-plaintext highlighter-rouge">devices</code> query, making it unique within the query set:</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">query</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">ams_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"ams"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">inventory_hostname</span><span class="p">:</span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="n">sin_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"sin"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">inventory_hostname</span><span class="p">:</span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="n">bkk_devices</span><span class="p">:</span><span class="n">devices</span><span class="p">(</span><span class="n">site</span><span class="p">:</span><span class="s2">"bkk"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">inventory_hostname</span><span class="p">:</span><span class="n">name</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Notice each <code class="language-plaintext highlighter-rouge">devices</code> query is aliased to <code class="language-plaintext highlighter-rouge">&lt;site name&gt;_devices</code> (<em>ams_devices</em>, <em>sin_devices</em>, and <em>bkk_devices</em>), making it unique within the query set. Aliasing solves the uniqueness problem within the multiple query set, making the request valid.</p> <p>Here are the returned results from Nautobot (snipped for brevity):</p> <div class="language-graphql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">data</span><span class="err">":</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">ams_devices</span><span class="err">":</span><span class="w"> </span><span class="err">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">edge</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">&lt;----</span><span class="w"> </span><span class="n">snip</span><span class="w"> </span><span class="err">----&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">ams</span><span class="err">-</span><span class="n">leaf</span><span class="err">-08"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">]</span><span class="p">,</span><span class="w"> </span><span class="err">"</span><span class="n">sin_devices</span><span class="err">":</span><span class="w"> </span><span class="err">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">sin</span><span class="err">-</span><span class="n">edge</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">sin</span><span class="err">-</span><span class="n">edge</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">sin</span><span class="err">-</span><span class="n">leaf</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">&lt;----</span><span class="w"> </span><span class="n">snip</span><span class="w"> </span><span class="err">----&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">sin</span><span class="err">-</span><span class="n">leaf</span><span class="err">-08"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">]</span><span class="p">,</span><span class="w"> </span><span class="err">"</span><span class="n">bkk_devices</span><span class="err">":</span><span class="w"> </span><span class="err">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">bkk</span><span class="err">-</span><span class="n">edge</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">bkk</span><span class="err">-</span><span class="n">edge</span><span class="err">-02"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">bkk</span><span class="err">-</span><span class="n">leaf</span><span class="err">-01"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="err">&lt;----</span><span class="w"> </span><span class="n">snip</span><span class="w"> </span><span class="err">----&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">"</span><span class="n">inventory_hostname</span><span class="err">":</span><span class="w"> </span><span class="err">"</span><span class="n">bkk</span><span class="err">-</span><span class="n">leaf</span><span class="err">-08"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Notice how each site’s <code class="language-plaintext highlighter-rouge">inventory_hostname</code> list is a value for the site’s alias, making this data programmatically easy to leverage.</p> <h2 id="graphql-efficiency">GraphQL Efficiency</h2> <p><em>Aliasing</em> is another tool GraphQL offers that allows the user to efficiently query for specific data with a minimal amount of requests and little to no required post-processing of the returned data.</p> <p>Be sure to check out the accompanying YouTube video for this post <a href="https://youtu.be/3T4cRsIqM7Y">here</a>, or click the image below.</p> <p><a href="https://youtu.be/3T4cRsIqM7Y"><img src="../../../static/images/blog_posts/graphQL/graphql_alias/yt_video_pic.png" alt="" /></a></p>Tim FiolaGraphQL aliasing is a very potent feature that allows the user to customize the query results by specifying key names in the returned data.