Jekyll2020-02-26T22:24:47+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comPython Objects2020-02-25T00:00:00+00:002020-02-25T00:00:00+00:00https://blog.networktocode.com/post/Python-Objects<p>“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the <code class="highlighter-rouge">str</code> class.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Creating a string variable </span><span class="o">&gt;&gt;&gt;</span> <span class="n">device_type</span> <span class="o">=</span> <span class="s">"switch"</span> <span class="c1"># device_type is an instance of the str class </span><span class="o">&gt;&gt;&gt;</span> <span class="nb">type</span><span class="p">(</span><span class="n">device_type</span><span class="p">)</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">str</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt;</span><span class="err"> </span></code></pre></div></div> <p>Classes themselves are instances of <code class="highlighter-rouge">type</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Creating a class object normally </span><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Cisco</span><span class="p">:</span> <span class="o">...</span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"cisco"</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">__main__</span><span class="o">.</span><span class="n">Cisco</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Cisco.vendor</span><span class="err"> </span><span class="s">'</span><span class="n">cisco</span><span class="s">'</span><span class="err"> </span><span class="s">&gt;&gt;&gt; isinstance(Cisco, type)</span><span class="err"> </span><span class="s">True</span><span class="err"> </span><span class="s"># Creating a class object explicitly from type</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista = type("Arista", (), {"vendor": "arista"})</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista</span><span class="err"> </span><span class="s">&lt;class '</span><span class="n">__main__</span><span class="o">.</span><span class="n">Arista</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista.vendor</span><span class="err"> </span><span class="s">'</span><span class="n">arista</span><span class="s">'</span><span class="err"> </span><span class="s">&gt;&gt;&gt;</span><span class="err"> </span></code></pre></div></div> <p>In addition to being an instance of a class, objects consist of attributes defining state and behavior. It is common to refer to state attributes as <strong>data attributes</strong> or <strong>variables</strong>, and behaviors are called <strong>methods</strong>. The Python builtin classes generally have many methods, but do not have data attributes. The <em>str</em> class provides methods for:</p> <ul> <li>Manipulating the string (capitalize, format, etc.)</li> <li>Finding information about the string (startswith, isnumeric, etc.)</li> <li>Segmenting the string (partition, split, etc.)</li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span> <span class="o">=</span> <span class="s">"this is a string"</span> <span class="c1"># Capitalize the first letter of each word </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">title</span><span class="p">()</span> <span class="s">'This Is A String'</span> <span class="c1"># Count the number of times the letter i appears </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="s">"i"</span><span class="p">)</span> <span class="mi">3</span> <span class="c1"># Convert the string into a list of words </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="p">[</span><span class="s">"this"</span><span class="p">,</span> <span class="s">"is"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="s">"string"</span><span class="p">]</span> </code></pre></div></div> <p>It is more common for custom classes to have data attributes that store information about the objects they imitate. For an object representing a switch, some example data attributes might be: <em>modules</em>, <em>uptime</em>, and <em>reachability</em>. Some example methods might be: <em>reboot</em>, and <em>ping</em>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">switch</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="n">hostname</span><span class="o">=</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="c1"># Example of a data attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">switch</span><span class="o">.</span><span class="n">uptime</span> <span class="s">'4 years, 3 days'</span> <span class="c1"># Example of a method attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">ping_stats</span> <span class="o">=</span> <span class="n">switch</span><span class="o">.</span><span class="n">ping</span><span class="p">(</span><span class="s">"10.1.1.1"</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="s">'pinging 10.1.1.1 times 5'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ping_stats</span> <span class="p">{</span> <span class="s">'attempts'</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s">'success'</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s">'fail'</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="s">'messages'</span><span class="p">:</span> <span class="p">[</span><span class="s">'pinged 10.1.1.1 in 0.21ms'</span><span class="p">,</span> <span class="s">'pinged 10.1.1.1 in 0.14ms'</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> <span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="data-attributes">Data Attributes</h2> <p>Data attributes are distinguished between class variables and instance variables. Class variables assign values that are the same for all objects of the class. Instance variables assign values for an instance of the class. Class variables are generally used for metadata fields, such as <code class="highlighter-rouge">vendor</code> in the first example. Instance variables contain more interesting information that the object methods act upon. Using the previous example, the <code class="highlighter-rouge">reachability</code> attribute might be used in a report to identify all switches that are unreachable. The Cisco class above is redefined to create instance variables upon initialization for hostname, connection status, and reachability information.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Cisco</span><span class="p">:</span> <span class="c1"># Class variable </span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"cisco"</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hostname</span><span class="p">):</span> <span class="c1"># Instance variables </span> <span class="bp">self</span><span class="o">.</span><span class="n">hostname</span> <span class="o">=</span> <span class="n">hostname</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">ping_reachability</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ping</span><span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="k">if</span> <span class="n">ping_reachability</span><span class="p">[</span><span class="s">"success"</span><span class="p">]:</span> <span class="bp">self</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"authorized"</span> <span class="k">else</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"unauthorized"</span> <span class="k">else</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"unreachable"</span> <span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">connected</span> <span class="k">else</span> <span class="bp">False</span> <span class="k">def</span> <span class="nf">ping</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">destination</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="p">{</span> <span class="s">"attempts"</span><span class="p">:</span> <span class="n">count</span><span class="p">,</span> <span class="s">"success"</span><span class="p">:</span> <span class="n">success_count</span><span class="p">,</span> <span class="s">"fail"</span><span class="p">:</span> <span class="n">fail_count</span><span class="p">,</span> <span class="s">"messages"</span><span class="p">:</span> <span class="p">[</span><span class="n">message</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">ping_results</span><span class="p">],</span> <span class="p">}</span> </code></pre></div></div> <p>Python uses dictionaries to store class and instance variables. Class variables share the same dictionary across all instances of the class, and instance variables are stored in a unique dictionary per instance. The class dict is stored in <code class="highlighter-rouge">&lt;class&gt;.__dict__</code>, and this is referenced on an instance with <code class="highlighter-rouge">&lt;obj&gt;.__class__.__dict__</code>. The instance dict is stored in <code class="highlighter-rouge">&lt;obj&gt;.__dict__</code>. Here are some examples showing how these two dictionaries behave.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># View the class dict </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">__dict__</span> <span class="n">mappingproxy</span><span class="p">({</span><span class="s">'vendor'</span><span class="p">:</span> <span class="s">'cisco'</span><span class="p">,</span> <span class="o">...</span><span class="p">})</span> <span class="c1"># Create two objects of the Cisco class </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="c1"># Viewing the instance dict </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">}</span> <span class="c1"># Reachability is an instance variable and can have different values across objects </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">reachability</span> <span class="s">'authorized'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">reachability</span> <span class="s">'unreachable'</span> <span class="c1"># The class dict is referenced within instances </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">__dict__</span> <span class="o">==</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__dict__</span> <span class="bp">True</span> <span class="c1"># Changing the vendor on the class is reflected on all instances </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">vendor</span> <span class="o">=</span> <span class="s">"CSCO"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'CSCO'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'CSCO'</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="attribute-precedence">Attribute Precedence</h2> <p>This distinction between class and instance variables is important for understanding Python’s attribute access behavior. The standard attribute lookup uses the following precedence until the attribute is found, or an Error is raised:</p> <ol> <li>Attributes defined on the instance</li> <li>Attributes defined on the class</li> <li>Attributes defined on inherited classes</li> </ol> <p>Practically, this means that overriding a class attribute on an instance will only affect the single instance and not other instances.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="c1"># rt1 and rt2 both have a value of "cisco" for the vendor attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="o">==</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="o">==</span> <span class="s">"cisco"</span> <span class="bp">True</span> <span class="c1"># rt1's instance dict does not have an attribute for `vendor` </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="o">=</span> <span class="s">"Cisco"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'Cisco'</span> <span class="c1"># rt1's instance dict was updated with a `vendor` attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">,</span> <span class="s">'vendor'</span><span class="p">:</span> <span class="s">'Cisco'</span><span class="p">}</span> <span class="c1"># rt2's vendor attribute remains unchanged </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'cisco'</span> </code></pre></div></div> <h2 id="mutable-attributes">Mutable Attributes</h2> <p>The above demonstrates the built-in safety provided when naming instance variables collides with class variables. However, this behavior might have surprising results when class variables point to mutable objects (such as lists). The Arista class is rewritten below to have a class variable defining the modules, with the value using a list for modular chassis.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Arista</span><span class="p">:</span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"arista"</span> <span class="n">modules</span> <span class="o">=</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">]</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">]</span> <span class="c1"># Appending to rt2's `mdoules` attribute affects rt1 </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">"7500E-72S"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">]</span> <span class="c1"># Using the `+=` operator on rt2's `modules` attributed affects rt1 </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span> <span class="o">+=</span> <span class="p">[</span><span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">,</span> <span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="c1"># Both modifications to rt2's `modules` attribute affect the class attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Arista</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">,</span> <span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="c1"># All references to `modules` are the same shared object </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Arista</span><span class="o">.</span><span class="n">modules</span> <span class="ow">is</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modulues</span> <span class="ow">is</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="conclusion">Conclusion</h2> <p>Understanding how objects work is critical to working with Python. Fundamental to working with objects is managing and using data attributes. Data attributes can be created on the class, or on each instance of the class. Instance level attributes have a higher precedence than class level attributes. Finally, when creating attributes on a class, it is important to consider how they will be used, and that mutable attributes can lead to unexpected behavior.</p> <p>-Jacob</p>Jacob McGill“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the str class.Backing up device configurations using Nornir and Ansible2020-02-20T00:00:00+00:002020-02-20T00:00:00+00:00https://blog.networktocode.com/post/nornir_ansible_backup<p>In this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but <strong>it depends :)</strong>. There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.</p> <p>Hopefully, the examples shown here can help you determine which tool best fits the need. I’m not going deep into the details on the different technologies or explaining all the components of each of the files, but I’ll provide some links to learn more about them if needed.</p> <h2 id="installation">Installation</h2> <p>If you want to follow along with the blog, Nornir and Ansible need to be installed in the machine locally or you will need to use tools that isolate dependencies like Docker and virtual environments. I ended up using Docker so if you have Docker installed in your machine and know how to use it, then feel free to use the <code class="highlighter-rouge">Dockerfile</code> below:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM python:3.7-stretch WORKDIR /ntc RUN apt-get update \ &amp;&amp; apt-get install tree RUN pip install black \ ansible \ nornir </code></pre></div></div> <p>After adding and saving the commands to the <code class="highlighter-rouge">Dockerfile</code> run the following commands to build the container image and start running the container.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build -t nornir . docker run -it --name nornir_ansible -v ${PWD}:/ntc nornir_ansible:latest /bin/bash </code></pre></div></div> <h2 id="backing-up-with-nornir">Backing up with Nornir</h2> <p><a href="https://nornir.readthedocs.io/en/latest/#welcome-to-nornir-s-documentation">Nornir</a> is an automation tool used to interact with networking devices. This tool is a bit different compared to other tools in the way that you write your own Python code to control the automation. For example, Ansible is written in Python but uses its own DSL which you use to describe what you want to have done. Nornir does not need a DSL instead, you can write everything in Python.</p> <h3 id="configuration-file">Configuration File</h3> <p>Now that everything has been installed, the first file to create is the Nornir configuration file. This file is similar to the <code class="highlighter-rouge">ansible.cfg</code> file which helps set all the system defaults, but in this case it’s going to be used to find the inventory file. Click on the following links to learn more about Nornir and Ansible configuration files:</p> <ul> <li><a href="https://nornir.readthedocs.io/en/stable/configuration/index.htmlhttps://nornir.readthedocs.io/en/stable/configuration/index.html">Nornir</a></li> <li><a href="https://docs.ansible.com/ansible/2.4/intro_configuration.html">Ansible</a></li> </ul> <p>Create a file called <code class="highlighter-rouge">nornir_config.yml</code> and inside the file store the following content:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="na">inventory</span><span class="pi">:</span> <span class="na">plugin</span><span class="pi">:</span> <span class="s">nornir.plugins.inventory.ansible.AnsibleInventory</span> <span class="na">options</span><span class="pi">:</span> <span class="na">hostsfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./inventory"</span> </code></pre></div></div> <blockquote> <p>Note: Since this container has access to the local machine, any tool or editors can be used with the same files the container is able to see. I’ll be using Visual Studio Code in my local machine.</p> </blockquote> <p>Inside the container run the <code class="highlighter-rouge">cat</code> command to make sure changes took place in the file.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>nornir_config.yml <span class="nt">---</span> inventory: plugin: nornir.plugins.inventory.ansible.AnsibleInventory options: hostsfile: <span class="s2">"./inventory"</span> </code></pre></div></div> <blockquote> <p>Note: The plugin used here <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/ansible.html">AnsibleInventory</a> allows us to source an inventory file that would normally be consumed by Ansible. There are other <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/index.html">plugins</a> that can be used like <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/simple.html">simple</a>, <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/nsot.html">NSOT</a> and <a href="https://nornir.readthedocs.io/en/stable/plugins/inventory/netbox.html#module-nornir.plugins.inventory.netbox">NetBox</a>.</p> </blockquote> <h3 id="inventory-file">Inventory File</h3> <p>The inventory file built here will be used for both Ansible and Nornir. The main differences here will be the hostvars names. For now Nornir will be using <code class="highlighter-rouge">hostname</code>, <code class="highlighter-rouge">platform</code>, <code class="highlighter-rouge">username</code> and <code class="highlighter-rouge">password</code>. Later, when the Ansible section comes, up a few other variables will be added. Create a file called <code class="highlighter-rouge">inventory</code> and place it in the same directory path as the <code class="highlighter-rouge">nornir_config.yml</code> file. Save it and run the <code class="highlighter-rouge">cat</code> command to make sure it saved properly and it’s seen by the container.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>inventory <span class="o">[</span>all] csr1 <span class="nb">hostname</span><span class="o">=</span>csr1 <span class="nv">platform</span><span class="o">=</span>ios <span class="nv">username</span><span class="o">=</span>ntc_user <span class="nv">password</span><span class="o">=</span>pass123 </code></pre></div></div> <p>Click on the following links to learn more about Nornir and Ansible inventory:</p> <ul> <li><a href="https://nornir.readthedocs.io/en/stable/tutorials/intro/inventory.html#Inventory">Nornir</a></li> <li><a href="https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#how-to-build-your-inventory">Ansible</a></li> </ul> <h3 id="backup-python-script">Backup Python Script</h3> <p>Now that the configuration and inventory file has been created it’s time to build the script that will backup the device configuration. The script will consist of functions that will build the necessary directory, backup files, and two different functions using the <code class="highlighter-rouge">netmiko</code> and <code class="highlighter-rouge">napalm</code> modules to backup the device configurations.</p> <p>The first step to building the script is to import all the needed libraries. In this case, the libraries needed are:</p> <ul> <li><a href="https://docs.python.org/3/library/os.html#module-os"><strong>os</strong></a>: This library is going to be used to help interact with the local environment and build the backup directory and file names for the configurations.</li> <li><a href="https://nornir.readthedocs.io/en/latest/ref/api/nornir.html#core"><strong>nornir</strong></a>: Nornir is an automation framework written in Python to be used with Python. <ul> <li><a href="https://nornir.readthedocs.io/en/latest/ref/api/nornir.html#initnornir"><strong>InitNornir</strong></a>: Object used to provide the configuration file containing system defaults. In this case, it’s being used to provide the inventory file.</li> <li><a href="https://nornir.readthedocs.io/en/latest/plugins/tasks/networking.html#nornir.plugins.tasks.networking.netmiko_send_command"><strong>netmiko_send_command</strong></a>: It’s a netmiko function used to send arbitrary commands to networking devices.</li> <li><a href="https://nornir.readthedocs.io/en/latest/plugins/tasks/networking.html#nornir.plugins.tasks.networking.napalm_get"><strong>napalm_get</strong></a>: Function that comes from the NAPALM library to get <code class="highlighter-rouge">facts</code> from the device. In this case, it will be used to retrieve the device configuration.</li> </ul> </li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span> <span class="kn">import</span> <span class="nn">logging</span> <span class="kn">from</span> <span class="nn">nornir</span> <span class="kn">import</span> <span class="n">InitNornir</span> <span class="kn">from</span> <span class="nn">nornir.plugins.tasks.networking</span> <span class="kn">import</span> <span class="n">netmiko_send_command</span><span class="p">,</span> <span class="n">napalm_get</span> </code></pre></div></div> <p>Two global variables need to be created. The first one, <code class="highlighter-rouge">BACKUP_DIR</code>, is the directory name used to store the backed up files and the second,<code class="highlighter-rouge">nr</code>, is used to carry the data stored in the <code class="highlighter-rouge">nornir_config.yml</code> file. In this case, it will contain the inventory file data.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">BACKUP_DIR</span> <span class="o">=</span> <span class="s">"backups/"</span> <span class="n">nr</span> <span class="o">=</span> <span class="n">InitNornir</span><span class="p">(</span><span class="n">config_file</span><span class="o">=</span><span class="s">"./nornir_config.yml"</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">create_backups_dir</code> is used to create a local directory called <code class="highlighter-rouge">backups</code> and it’s using the <code class="highlighter-rouge">os</code> library to create a directory with the name stored in the <code class="highlighter-rouge">BACKUP_DIR</code> variable.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">create_backups_dir</span><span class="p">():</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">):</span> <span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">save_config_to_file</code> function is used to create the backed up device configuration files inside the <code class="highlighter-rouge">backups</code> directory. Notice the parameters used will take as input <code class="highlighter-rouge">method</code>, <code class="highlighter-rouge">hostname</code> and <code class="highlighter-rouge">config</code>, these parameters will get their data from the <code class="highlighter-rouge">get_netmiko_backups()</code> and <code class="highlighter-rouge">get_napalm_backups()</code> functions that will be created in the next few steps.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">save_config_to_file</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="p">):</span> <span class="n">filename</span> <span class="o">=</span> <span class="n">f</span><span class="s">"{hostname}-{method}.cfg"</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">,</span> <span class="n">filename</span><span class="p">),</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">config</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">get_netmiko_backups</code> function will run the <code class="highlighter-rouge">netmiko_send_command</code> module to send a <strong>“show run”</strong> command to the network device stored in the <code class="highlighter-rouge">nr</code> variable and it will be stored in the <code class="highlighter-rouge">backups_results</code> variable.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_netmiko_backups</span><span class="p">():</span> <span class="n">backup_results</span> <span class="o">=</span> <span class="n">nr</span><span class="o">.</span><span class="n">run</span><span class="p">(</span> <span class="n">task</span><span class="o">=</span><span class="n">netmiko_send_command</span><span class="p">,</span> <span class="n">command_string</span><span class="o">=</span><span class="s">"show run"</span> <span class="p">)</span> <span class="k">for</span> <span class="n">hostname</span> <span class="ow">in</span> <span class="n">backup_results</span><span class="p">:</span> <span class="n">save_config_to_file</span><span class="p">(</span> <span class="n">method</span><span class="o">=</span><span class="s">"netmiko"</span><span class="p">,</span> <span class="n">hostname</span><span class="o">=</span><span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">backup_results</span><span class="p">[</span><span class="n">hostname</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">result</span><span class="p">,</span> <span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">backups_results</code> variable will end up with a value that stores the hostname and configuration as a result looking something like this:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; print(backup_results) AggregatedResult (netmiko_send_command): {'csr1': MultiResult: [Result: "netmiko_send_command"]} &gt;&gt;&gt; &gt;&gt;&gt; </code></pre></div></div> <blockquote> <p>Note: <code class="highlighter-rouge">AggregatedResult</code> is a dict-like object that aggregates the results for all devices. You can access each individual result by doing <code class="highlighter-rouge">my_aggr_result["hostname_of_device"]</code></p> </blockquote> <p>The data inside <code class="highlighter-rouge">backups_results</code> can be extracted by using a for loop and stored in <code class="highlighter-rouge">hostname</code>. The data iterated through with the for loop will return the hostname information and backup configuration.</p> <p>The <code class="highlighter-rouge">get_napalm_backups</code> function will be built the same way, except this time it will be using the <code class="highlighter-rouge">napalm_get</code> module to extract the device configuration.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_napalm_backups</span><span class="p">():</span> <span class="n">backup_results</span> <span class="o">=</span> <span class="n">nr</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">task</span><span class="o">=</span><span class="n">napalm_get</span><span class="p">,</span> <span class="n">getters</span><span class="o">=</span><span class="p">[</span><span class="s">"config"</span><span class="p">])</span> <span class="k">for</span> <span class="n">hostname</span> <span class="ow">in</span> <span class="n">backup_results</span><span class="p">:</span> <span class="n">config</span> <span class="o">=</span> <span class="n">backup_results</span><span class="p">[</span><span class="n">hostname</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">result</span><span class="p">[</span><span class="s">"config"</span><span class="p">][</span><span class="s">"startup"</span><span class="p">]</span> <span class="n">save_config_to_file</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s">"napalm"</span><span class="p">,</span> <span class="n">hostname</span><span class="o">=</span><span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">)</span> </code></pre></div></div> <p>The last function <code class="highlighter-rouge">main()</code> will execute all the functions and the code entry point is added to start the script when executed.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="n">create_backups_dir</span><span class="p">()</span> <span class="n">get_netmiko_backups</span><span class="p">()</span> <span class="n">get_napalm_backups</span><span class="p">()</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </code></pre></div></div> <p>Once the script has been built, execute the command <code class="highlighter-rouge">black</code> to clean up and make sure all formatting is correct and use the command <code class="highlighter-rouge">cat</code> to make sure everything has been saved correctly. Click on the following link to learn more about <a href="https://pypi.org/project/black/"><strong>black</strong></a>.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# black backups.py All <span class="k">done</span><span class="o">!</span> ✨ 🍰 ✨ 1 file left unchanged. </code></pre></div></div> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@6b91330f9674:/ntc# <span class="nb">cat </span>backups.py import os from nornir import InitNornir from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get BACKUP_DIR <span class="o">=</span> <span class="s2">"backups/"</span> nr <span class="o">=</span> InitNornir<span class="o">(</span><span class="nv">config_file</span><span class="o">=</span><span class="s2">"./nornir_config.yml"</span><span class="o">)</span> def create_backups_dir<span class="o">()</span>: <span class="k">if </span>not os.path.exists<span class="o">(</span>BACKUP_DIR<span class="o">)</span>: os.mkdir<span class="o">(</span>BACKUP_DIR<span class="o">)</span> def save_config_to_file<span class="o">(</span>method, <span class="nb">hostname</span>, config<span class="o">)</span>: filename <span class="o">=</span> f<span class="s2">"{hostname}-{method}.cfg"</span> with open<span class="o">(</span>os.path.join<span class="o">(</span>BACKUP_DIR, <span class="nb">hostname</span><span class="o">)</span>, <span class="s2">"w"</span><span class="o">)</span> as f: f.write<span class="o">(</span>config<span class="o">)</span> def get_netmiko_backups<span class="o">()</span>: backup_results <span class="o">=</span> nr.run<span class="o">(</span> <span class="nv">task</span><span class="o">=</span>netmiko_send_command, <span class="nv">command_string</span><span class="o">=</span><span class="s2">"show run"</span> <span class="o">)</span> <span class="k">for </span><span class="nb">hostname </span><span class="k">in </span>backup_results: save_config_to_file<span class="o">(</span> <span class="nv">method</span><span class="o">=</span><span class="s2">"netmiko"</span>, <span class="nb">hostname</span><span class="o">=</span><span class="nb">hostname</span>, <span class="nv">config</span><span class="o">=</span>backup_results[hostname][0].result, <span class="o">)</span> def get_napalm_backups<span class="o">()</span>: backup_results <span class="o">=</span> nr.run<span class="o">(</span><span class="nv">task</span><span class="o">=</span>napalm_get, <span class="nv">getters</span><span class="o">=[</span><span class="s2">"config"</span><span class="o">])</span> <span class="k">for </span><span class="nb">hostname </span><span class="k">in </span>backup_results: config <span class="o">=</span> backup_results[hostname][0].result[<span class="s2">"config"</span><span class="o">][</span><span class="s2">"startup"</span><span class="o">]</span> save_config_to_file<span class="o">(</span><span class="nv">method</span><span class="o">=</span><span class="s2">"napalm"</span>, <span class="nb">hostname</span><span class="o">=</span><span class="nb">hostname</span>, <span class="nv">config</span><span class="o">=</span>config<span class="o">)</span> def main<span class="o">()</span>: create_backups_dir<span class="o">()</span> get_netmiko_backups<span class="o">()</span> get_napalm_backups<span class="o">()</span> <span class="k">if </span>__name__ <span class="o">==</span> <span class="s2">"__main__"</span>: main<span class="o">()</span> </code></pre></div></div> <h3 id="running-the-python-script">Running the Python Script</h3> <p>Now that the script is complete, it’s time to execute it by using the <code class="highlighter-rouge">Python</code> command and file name right next to it <code class="highlighter-rouge">python backups.py</code>.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# python backups.py root@c2e446b364a8:/ntc# </code></pre></div></div> <p>The <code class="highlighter-rouge">tree</code> command can be used to view the configuration files stored in the directory <code class="highlighter-rouge">backups</code> configuration files stored inside of it with the specified file names built in the <code class="highlighter-rouge">save_config_to_file</code> function.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── inventory ├── nornir.log └── nornir_config.yml 1 directory, 7 files </code></pre></div></div> <blockquote> <p>Note: Take a look at the backed up configuration files using the <code class="highlighter-rouge">cat</code> command or an editor of choice. They are left off from this as they take up a lot of screen space.</p> </blockquote> <p>By default, Nornir automatically configures logging when InitNornir is called and a file is created locally that can be viewed what functions where ran that belong to the <code class="highlighter-rouge">Nornir</code> library.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>nornir.log 2020-01-02 21:33:26,519 - nornir.core - INFO - run<span class="o">()</span> - Running task <span class="s1">'netmiko_send_command'</span> with args <span class="o">{</span><span class="s1">'command_string'</span>: <span class="s1">'show run'</span><span class="o">}</span> on 1 hosts 2020-01-02 21:33:32,134 - nornir.core - INFO - run<span class="o">()</span> - Running task <span class="s1">'napalm_get'</span> with args <span class="o">{</span><span class="s1">'getters'</span>: <span class="o">[</span><span class="s1">'config'</span><span class="o">]}</span> on 1 hosts </code></pre></div></div> <h2 id="backing-up-with-ansible">Backing up with Ansible</h2> <p>Ansible is a configuration management tool originally created to interact with Linux servers, eventually it started gaining traction in the networking industry to manage networking device configurations. Ansible was built using Python but it uses YAML to configure all the automation tasks. The idea is to simplify the work that it takes to start automating.</p> <h3 id="configuration-file-1">Configuration file</h3> <p>The <code class="highlighter-rouge">ansible.cfg</code> configuration file is used to set some of the system default values. Create the <code class="highlighter-rouge">ansible.cfg</code> file in the root of the directory next to all the other files created previously. Then use the <code class="highlighter-rouge">cat</code> command to make sure everything was stored correctly.</p> <ul> <li><strong>inventory</strong>: Sets the location of the inventory file, by default it will look for it in <code class="highlighter-rouge">etc/ansible/hosts</code> or while running the playbook the <code class="highlighter-rouge">-i</code> flag can be used to point to the location of the file. In this case, it is pointed to the inventory file we will create in the next step within the same directory.</li> <li><strong>host_key_checking</strong>: This has been disabled to prevent host key checking and prevent issues when running the playbook.</li> <li><strong>interpreter_python</strong>: Has been set in the <code class="highlighter-rouge">ansible.cfg</code> file to prevent a new warning added to the new version of Ansible and specify what interpreter to use.</li> <li><strong>gathering</strong>: Is set to smart to gather facts by default, but don’t regather if already gathered.</li> </ul> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>ansible.cfg <span class="o">[</span>defaults] inventory <span class="o">=</span> ./inventory host_key_checking <span class="o">=</span> False interpreter_python <span class="o">=</span> /usr/bin/python gathering <span class="o">=</span> smart </code></pre></div></div> <h3 id="inventory">Inventory</h3> <p>Edit the same inventory file that was used for Nornir, except this time add the following variables for Ansible.</p> <ul> <li><code class="highlighter-rouge">ansible_user=ntc_user</code></li> <li><code class="highlighter-rouge">ansible_password=pass123</code></li> <li><code class="highlighter-rouge">ansible_network_os=ios</code></li> </ul> <p>Again use the <code class="highlighter-rouge">cat</code> command to make sure the changes took effect.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>inventory <span class="o">[</span>all] csr1 <span class="nb">hostname</span><span class="o">=</span>csr1 <span class="nv">platform</span><span class="o">=</span>ios <span class="nv">username</span><span class="o">=</span>ntc_user <span class="nv">password</span><span class="o">=</span>pass123 <span class="nv">ansible_user</span><span class="o">=</span>ntc_user <span class="nv">ansible_password</span><span class="o">=</span>pass123 <span class="nv">ansible_network_os</span><span class="o">=</span>ios </code></pre></div></div> <h3 id="playbook">Playbook</h3> <p>The first part to the Playbook is the play definition. The play definition will contain a list of keys such as:</p> <ul> <li><strong>name</strong>: Any arbitrary name for the Playbook.</li> <li><strong>hosts</strong>: The devices that will be targeted in the scope of the play.</li> <li><strong>connection</strong>: Type of connection. In this case <code class="highlighter-rouge">network_cli</code> which is basically SSH.</li> <li><strong>tasks</strong>: The key that contains all the automation steps that belong to the play.</li> <li><strong>gather_facts</strong>: Before Ansible version 2.9, this key was normally disabled when interacting with networking devices. Now, it has the ability to gather device facts at the play definition level rather than at the task level.</li> </ul> <p>The second part of the Playbook nested under the <code class="highlighter-rouge">tasks</code> key will have a list of <code class="highlighter-rouge">key:value</code> pair tasks.</p> <ul> <li><strong>name</strong>: Optional but recommended key that can have any arbitrary name for the task.</li> <li><a href="https://docs.ansible.com/ansible/latest/modules/ios_config_module.html#ios-config-manage-cisco-ios-configuration-sections"><strong>ios_config</strong></a>: This is the Python module used to run configuration commands against <code class="highlighter-rouge">ios</code> devices.</li> <li><a href="https://docs.ansible.com/ansible/latest/modules/ios_config_module.html#parameters"><strong>backup</strong></a>: Module parameter used to backup the device configuration. By default, if the backup directory is not created then it will create one in the same location as the Ansible playbook and store the configuration files in the backup directory.</li> </ul> <blockquote> <p>Note: A recent parameter was added as <code class="highlighter-rouge">backup_options</code> with sub-options to change the directory path rather than the default which is <code class="highlighter-rouge">backup</code> and the ability to change the filename of the device configuration. In this case, the default value is set with this format <code class="highlighter-rouge">&lt;hostname&gt;_config.&lt;current-date&gt;@&lt;current-time&gt;</code></p> </blockquote> <p>Create the Ansible Playbook and call it <code class="highlighter-rouge">backup_playbook.yml</code>. Use the <code class="highlighter-rouge">cat</code> command to make sure the content is stored correctly:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>backup_playbook.yml <span class="nt">---</span> - name: BACKUP PLAYBOOK hosts: csr1 connection: network_cli gather_facts: <span class="nb">yes </span>tasks: - name: BACKUP USING IOS_CONFIG ios_config: backup: <span class="nb">yes</span> </code></pre></div></div> <h3 id="running-the-playbook">Running the Playbook</h3> <p>To run the Ansible Playbook use the command <code class="highlighter-rouge">ansible-playbook backup_playbook.yml</code>. The output is shown below:</p> <blockquote> <p>Note: There is an extra task that says <code class="highlighter-rouge">Gathering Facts</code> which is a new feature added to Ansible that now gathers facts from networking devices at the play definition level. Take a look at this blog <a href="https://blog.networktocode.com/post/ansible-gather_facts-enabled/">Ansible gather_facts for Networking Devices</a> to learn a little more about it.</p> </blockquote> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml PLAY <span class="o">[</span>BACKUP PLAYBOOK] <span class="k">**********************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">**********************************************************************************</span> <span class="o">[</span>WARNING]: Ignoring <span class="nb">timeout</span><span class="o">(</span>10<span class="o">)</span> <span class="k">for </span>ios_facts ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_CONFIG] <span class="k">***************************************************************************</span> changed: <span class="o">[</span>csr1] PLAY RECAP <span class="k">***********************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>2 <span class="nv">changed</span><span class="o">=</span>1 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>Use the tree command to view the content inside and notice how the device configuration is stored in the <code class="highlighter-rouge">backup</code> directory:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── ansible.cfg ├── backup │   └── csr1_config.2020-01-03@15:43:16 ├── backup_playbook.yml ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── inventory ├── nornir.log └── nornir_config.yml 2 directories, 10 files </code></pre></div></div> <h3 id="backing-up-with-ios_facts">Backing up with ios_facts</h3> <p>There’s another way of backing up the device configurations by using <code class="highlighter-rouge">ios_facts</code>, except this time rather than creating a new task to gather facts we can leverage the gathering of facts that takes place at the play definition level and add a task that will create a new directory to store the device configurations and another task to copy the content from <code class="highlighter-rouge">ansible_facts</code> as a configuration file.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP PLAYBOOK</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">csr1</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">yes</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP USING IOS_CONFIG</span> <span class="na">ios_config</span><span class="pi">:</span> <span class="na">backup</span><span class="pi">:</span> <span class="s">yes</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CREATE BACKUP DIRECTORY</span> <span class="na">file</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">./ansible_backup</span> <span class="na">state</span><span class="pi">:</span> <span class="s">directory</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP USING IOS_FACTS</span> <span class="na">copy</span><span class="pi">:</span> <span class="na">content</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_net_config</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">dest</span><span class="pi">:</span> <span class="s">./ansible_backup/{{ inventory_hostname }}_config.{{ now() }}.cfg</span> </code></pre></div></div> <p>In the playbook above, the <code class="highlighter-rouge">copy</code> module is used to copy content stored in the <code class="highlighter-rouge">ansible_net_config</code> key that is found inside <code class="highlighter-rouge">ansible_facts</code>. Any <code class="highlighter-rouge">net_*</code> key under <code class="highlighter-rouge">ansible_facts</code> can be accessed directly by using the <code class="highlighter-rouge">ansible_net_*</code> key rather than accessing the nested data structure. Now the new tasks have been added the playbook can be run again.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml PLAY <span class="o">[</span>BACKUP PLAYBOOK] <span class="k">**********************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">**********************************************************************************</span> <span class="o">[</span>WARNING]: Ignoring <span class="nb">timeout</span><span class="o">(</span>10<span class="o">)</span> <span class="k">for </span>ios_facts ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_CONFIG] <span class="k">***************************************************************************</span> changed: <span class="o">[</span>csr1] TASK <span class="o">[</span>CREATE BACKUP DIRECTORY] <span class="k">****************************************************************************</span> changed: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_FACTS] <span class="k">*****************************************************************************</span> changed: <span class="o">[</span>csr1] PLAY RECAP <span class="k">*************************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>3 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>Notice how every time the first task is run a new backup configuration is created since it’s been run with a different time stamp. The same can be done with the <code class="highlighter-rouge">ios_facts</code> backup method.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── ansible.cfg ├── ansible_backup │   └── csr1_config.2020-01-03 19:01:43.277008.cfg ├── backup │   ├── csr1_config.2020-01-03@15:43:16 │   └── csr1_config.2020-01-03@19:01:42 ├── backup_playbook.yml ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── csr1_config.2020-01-03@15:44:07 ├── inventory ├── nornir.log └── nornir_config.yml 3 directories, 13 files </code></pre></div></div> <h3 id="summary">Summary</h3> <p>There are many automation tools and modules that can help automate the backup of device configurations. The ones shown in this blog are some of the basic ways to do it, which ever method you choose will do the job. There can be different modifications that can be done to the Nornir script or even the Ansible playbook to better fit your application and hopefully what I have shown here can give you a start on what tool to use.</p> <p>-Hector</p>Hector IsazaIn this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but it depends :). There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.My first REST API work with Python2020-02-11T00:00:00+00:002020-02-11T00:00:00+00:00https://blog.networktocode.com/post/Basic-API-use-with-python<p>I come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.</p> <p>One of the hardest parts of writing this was choosing an API to discuss. Some of the things I really wanted from an API were:</p> <ul> <li>A Swagger page: I always thought of this as a webpage that has the API laid out nicely and gives the opportunity to run the API calls directly on it. This is more technically known as an Open API spec.</li> <li>A platform that was common in the networking world that people would be likely to run across.</li> <li>A platform that was easily available and didn’t require you to build a new system.</li> </ul> <p>Sadly, this combo doesn’t seem to exist. In the end, I decided to go over 2 APIs:</p> <ul> <li>NetBox <ul> <li>“An open source web application designed to help manage and document computer networks.”</li> <li>Documentation that is easy for a beginner with its Swagger page.</li> <li>https://netbox.readthedocs.io/en/stable/</li> </ul> </li> <li>ACI <ul> <li>A Network policy-based automation model for data centers.</li> <li>Cisco DevNet has an <a href="https://devnetsandbox.cisco.com/RM/Diagram/Index/5a229a7c-95d5-4cfd-a651-5ee9bc1b30e2?diagramType=Topology">always on sandbox</a> where you can access a working ACI environment at any time.</li> <li>When I was working on this API about a year ago, I found the documentation hard to navigate. I can see how it would work well for someone with more experience, but it didn’t work well for me at the time, and I haven’t kept up to date to know if their documentation has gotten more friendly.</li> <li>https://learningnetwork.cisco.com/docs/DOC-32331</li> </ul> </li> </ul> <h2 id="rest-api-core-concepts">REST API Core Concepts</h2> <p>While I’m not going to go to in depth, I want to walk through a few examples to give you the general idea of how a REST API works.</p> <p>There are two main methods of authentication. One where the API is provided a username and password and the system gives back a temporary token. This process is often used in systems where you are running a system such as ACI. In the other method, a token you generate from a UI serves as your authentication. These are often used for web applications such as Slack, Twitter, Facebook, and fortunately NetBox, so we can see how to do that.</p> <p>The general idea behind the REST API is you are going to an URL rather than typing in commands. Let’s look at this like a Cisco CLI. For example, your command might be <code class="highlighter-rouge">show run interface gi1</code>. The equivalent URL might be something like <code class="highlighter-rouge">device.com/api/conf/interface/gi1</code> so to work with <code class="highlighter-rouge">gi2</code> the URL would just change <code class="highlighter-rouge">device.com/api/conf/interface/gi2</code>.</p> <p>The main types of calls are below.</p> <ul> <li><code class="highlighter-rouge">GET</code> <ul> <li>Pulling data.</li> <li>You can’t really cause an issue doing this as long as you don’t pull too much at one time, e.g. pulling 1,000,000 records may bog down the system.</li> <li>Equivalent of a show command.</li> </ul> </li> <li><code class="highlighter-rouge">PUT/POST/PATCH</code> <ul> <li>Changing or creating data.</li> <li>Equivalent of configuration changes.</li> </ul> </li> <li><code class="highlighter-rouge">DELETE</code> <ul> <li>Delete data.</li> </ul> </li> </ul> <p>With the same examples from above, we will do a <code class="highlighter-rouge">GET</code> on <code class="highlighter-rouge">device.com/api/conf/interface/gi1</code>. It returns the dictionary below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="n">description</span><span class="p">:</span> <span class="s">"Disconnected"</span> <span class="n">status</span><span class="p">:</span> <span class="s">"shut"</span> <span class="n">mode</span><span class="p">:</span> <span class="s">"access"</span> <span class="n">vlans</span><span class="p">:</span> <span class="mi">110</span> <span class="p">}</span> </code></pre></div></div> <p>We now know the description on the interface, the VLAN, and that it is shut down. Let’s say we wanted to make a change, just update it as below and <code class="highlighter-rouge">PUT</code> it to the same URL.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="n">description</span><span class="p">:</span> <span class="s">"VM Server Bob"</span> <span class="n">status</span><span class="p">:</span> <span class="s">"no shut"</span> <span class="n">mode</span><span class="p">:</span> <span class="s">"trunk"</span> <span class="n">vlans</span><span class="p">:</span> <span class="s">"10-100"</span> <span class="p">}</span> </code></pre></div></div> <p>The port is now up, with a new description, as a trunk allowing VLANs 10-100 though. While not quite as easy to understand the url above this URL <code class="highlighter-rouge">https://device.com/restconf/data/Cisco-IOS-XE-native:native/interface=GigabitEthernet/1</code> would work for the RESTCONF API for IOS-XE devices and has an easy to read URL.</p> <h2 id="netbox">NetBox</h2> <p>I am running an instance of NetBox in my lab, the IP is 192.168.0.159, so that’s what all my examples will be of. It’s not hardened or using https, so all of my examples are using http on port 8000. Your box may vary, adjust accordingly.</p> <p>First, let’s build and grab a token from <code class="highlighter-rouge">http://192.168.0.159:8000/user/api-tokens/</code>.</p> <center><img src="../../../static/images/basic_api_use_with_python/Netbox_get_token.jpg" /></center> <p>Next, we’ll look at the swagger documentation page at <code class="highlighter-rouge">http://192.168.0.159:8000/api/docs/</code>. Login to the session</p> <center><img src="../../../static/images/basic_api_use_with_python/authorize_swagger.jpg" /></center> <center><img src="../../../static/images/basic_api_use_with_python/paste_in_token.jpg" /></center> <p>and find an action to do, (e.g. get a device from DCIM,) select that, then select “Try it out.”</p> <center><img src="../../../static/images/basic_api_use_with_python/try_it_out.jpg" /></center> <p>We can pull all devices if we were to look at another option, but the API call gives us lots of options. A screenshot would be overly busy, so I am just going to pull one device in this example. Please note the required tag.</p> <center><img src="../../../static/images/basic_api_use_with_python/execute.jpg" /></center> <p>The API call gives us all the data for that we asked for. Please pay special note to the cURL script. That is the call it made to the NetBox API. It sent a <code class="highlighter-rouge">GET</code> request to <code class="highlighter-rouge">http://192.168.0.159:8000/api/dcim/devices/5/</code> please note the initial line we chose was <code class="highlighter-rouge">/dcim/devices/{id}/</code></p> <center><img src="../../../static/images/basic_api_use_with_python/gives_all_data.jpg" /></center> <p>The cURL script is running an HTTP request to the URL listed, and you could run the same request on nearly all systems. For example, right now you can run “curl http://www.google.com” and make the HTTP call to Google. One of the really nice things about that cURL script is we can go to https://curl.trillworks.com, paste in the cURL script, and it will give us the Python equivalent. NOTE that if the request has the token in it you should be careful. Swap out the token before posting it to the website unless you like giving admin credentials to strangers.</p> <center><img src="../../../static/images/basic_api_use_with_python/gives_python_code.jpg" /></center> <p>Honestly there’s not much to be scared of here. Just working though Swagger and putting a few functions together from what you get from the website. Super easy… Now let’s look at ACI.</p> <h2 id="aci">ACI</h2> <p>In this example, the Cisco DevNet ACI emulator will be used. Set the username, password, and the base URL. This URL will be in every call made. To compare to the NetBox example, the base_url would have been <code class="highlighter-rouge">http://192.168.0.159:8000/api</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">username</span> <span class="o">=</span> <span class="s">"admin"</span> <span class="n">password</span> <span class="o">=</span> <span class="s">"ciscopsdt"</span> <span class="n">base_url</span> <span class="o">=</span> <span class="s">"https://sandboxapicdc.cisco.com/api/"</span> </code></pre></div></div> <p>The login script is honestly one of the most difficult parts of the whole process, as documentation is often sparse and there is no industry standard for field names across REST APIs. The process involves simply logging in, getting a token, and returning it. This is how many APIs work, and there are generally small differences for every API. Most of this was from sample code obtained from DevNet a few years ago.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_token</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">base_url</span><span class="p">):</span> <span class="c1"># Disable warning messages that come with using self signed certs </span> <span class="n">requests</span><span class="o">.</span><span class="n">packages</span><span class="o">.</span><span class="n">urllib3</span><span class="o">.</span><span class="n">disable_warnings</span><span class="p">()</span> <span class="c1"># create credentials structure </span> <span class="n">name_pwd</span> <span class="o">=</span> <span class="p">{</span><span class="s">"aaaUser"</span><span class="p">:</span> <span class="p">{</span><span class="s">"attributes"</span><span class="p">:</span> <span class="p">{</span><span class="s">"name"</span><span class="p">:</span> <span class="n">username</span><span class="p">,</span> <span class="s">"pwd"</span><span class="p">:</span> <span class="n">password</span><span class="p">}}}</span> <span class="n">json_credentials</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">name_pwd</span><span class="p">)</span> <span class="c1"># Create the full URL we need in order to login </span> <span class="c1"># You'll find we do this a lot </span> <span class="n">login_url</span> <span class="o">=</span> <span class="n">base_url</span> <span class="o">+</span> <span class="s">"aaaLogin.json"</span> <span class="c1"># log in to API </span> <span class="c1"># requests is a library that can do HTTP transactions, in this case we POSTing up (Telling it) our </span> <span class="c1"># credentials to get the token back </span> <span class="c1"># Verify=False ignores self signed certs, but would kick off errors if we hadn't disabled them up above </span> <span class="n">post_response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">login_url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json_credentials</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="c1"># getting the token from login response structure </span> <span class="n">auth</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">post_response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="n">login_attributes</span> <span class="o">=</span> <span class="n">auth</span><span class="p">[</span><span class="s">"imdata"</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">"aaaLogin"</span><span class="p">][</span><span class="s">"attributes"</span><span class="p">]</span> <span class="n">auth_token</span> <span class="o">=</span> <span class="n">login_attributes</span><span class="p">[</span><span class="s">"token"</span><span class="p">]</span> <span class="c1"># create token dictionary that we will use for authentication from here on out for auth cookie </span> <span class="n">cookies</span> <span class="o">=</span> <span class="p">{}</span> <span class="n">cookies</span><span class="p">[</span><span class="s">"APIC-Cookie"</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth_token</span> <span class="k">return</span> <span class="n">cookies</span> </code></pre></div></div> <p>Let’s review the documentation for the ACI API. Once again, this is how it worked back in late 2018, the documentation may have changed since. Access the API inspector,</p> <center><img src="../../../static/images/basic_api_use_with_python/get_api_inspector.jpg" /></center> <p>then proceed to the interesting area in the GUI, paying close attention to your last click. The example will return with the switch information, so I went to <code class="highlighter-rouge">Fabric (top tap) -&gt; Inventory -&gt; POD1 (expanded). Sadly, the URL alone doesn't always give you exactly what you need. In this example I had to cut off the </code>&amp;subscription=yes` section. Often, determinng the URL can be the hardest part.</p> <center><img src="../../../static/images/basic_api_use_with_python/find_call.jpg" /></center> <p>You can also see that the URL is asking for JSON in the <code class="highlighter-rouge">pod-1.json?</code> section. For ACI, it will return either XML or JSON. Personally, I find JSON easier to deal with in Python. Using the JSON library it’s not hard to convert JSON to a dictionary; <code class="highlighter-rouge">https://www.w3schools.com/python/python_json.asp</code> does a better job at explaining the details. Let’s take a look at this process.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span> <span class="k">def</span> <span class="nf">pull_pod1</span><span class="p">(</span><span class="n">cookies</span><span class="p">,</span> <span class="n">base_url</span><span class="p">):</span> <span class="n">sensor_url</span> <span class="o">=</span> <span class="p">(</span> <span class="n">base_url</span> <span class="o">+</span> <span class="s">'/node/mo/topology/pod-1.json?query-target=children&amp;target-subtree-class=fabricNode&amp;query-target-filter=and(not(wcard(fabricNode.dn,</span><span class="si">%22</span><span class="s">__ui_</span><span class="si">%22</span><span class="s">)),and(ne(fabricNode.role,"controller")))'</span> <span class="p">)</span> <span class="n">get_response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">sensor_url</span><span class="p">,</span> <span class="n">cookies</span><span class="o">=</span><span class="n">cookies</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="k">return</span> <span class="n">get_response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span> <span class="n">username</span> <span class="o">=</span> <span class="s">"admin"</span> <span class="n">password</span> <span class="o">=</span> <span class="s">"ciscopsdt"</span> <span class="n">base_url</span> <span class="o">=</span> <span class="s">"https://sandboxapicdc.cisco.com/api/"</span> <span class="c1">## Calling the authentication function above to get the auth cookie </span><span class="n">cookies</span> <span class="o">=</span> <span class="n">get_token</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">base_url</span><span class="p">)</span> <span class="n">pprint</span><span class="p">(</span><span class="n">pull_pod1</span><span class="p">(</span><span class="n">cookies</span><span class="p">,</span> <span class="n">base_url</span><span class="p">))</span> </code></pre></div></div> <p>This gives us all the devices in POD1. I’ll just be putting in one device for the sake of space</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="s">"imdata"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"fabricNode"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"attributes"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"adSt"</span><span class="p">:</span> <span class="s">"on"</span><span class="p">,</span> <span class="s">"address"</span><span class="p">:</span> <span class="s">"10.0.112.64"</span><span class="p">,</span> <span class="s">"annotation"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"apicType"</span><span class="p">:</span> <span class="s">"apic"</span><span class="p">,</span> <span class="s">"childAction"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"delayedHeartbeat"</span><span class="p">:</span> <span class="s">"no"</span><span class="p">,</span> <span class="s">"dn"</span><span class="p">:</span> <span class="s">"topology/pod-1/node-101"</span><span class="p">,</span> <span class="s">"extMngdBy"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"fabricSt"</span><span class="p">:</span> <span class="s">"active"</span><span class="p">,</span> <span class="s">"id"</span><span class="p">:</span> <span class="s">"101"</span><span class="p">,</span> <span class="s">"lastStateModTs"</span><span class="p">:</span> <span class="s">"2019-12-12T14:22:14.238+00:00"</span><span class="p">,</span> <span class="s">"lcOwn"</span><span class="p">:</span> <span class="s">"local"</span><span class="p">,</span> <span class="s">"modTs"</span><span class="p">:</span> <span class="s">"2019-12-12T14:23:02.315+00:00"</span><span class="p">,</span> <span class="s">"model"</span><span class="p">:</span> <span class="s">"N9K-C9396PX"</span><span class="p">,</span> <span class="s">"monPolDn"</span><span class="p">:</span> <span class="s">"uni/fabric/monfab-default"</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"leaf-1"</span><span class="p">,</span> <span class="s">"nameAlias"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"nodeType"</span><span class="p">:</span> <span class="s">"unspecified"</span><span class="p">,</span> <span class="s">"role"</span><span class="p">:</span> <span class="s">"leaf"</span><span class="p">,</span> <span class="s">"serial"</span><span class="p">:</span> <span class="s">"TEP-1-101"</span><span class="p">,</span> <span class="s">"status"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"uid"</span><span class="p">:</span> <span class="s">"0"</span><span class="p">,</span> <span class="s">"vendor"</span><span class="p">:</span> <span class="s">"Cisco Systems, Inc"</span><span class="p">,</span> <span class="s">"version"</span><span class="p">:</span> <span class="s">""</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> </code></pre></div></div> <blockquote> <p>Note: Output omitted and modified for the sake of brevity and clarity.</p> </blockquote> <p>You can observe several key pieces of information about this switch. The DN info, ID, model number, serial number, etc. With ACI, I often had to pull one piece of data so that I could understand how to pull another piece of data. Finding the DN was almost always an important step in my programs.</p> <p>There is an easier, albeit less stable way (based on my observations) to find the URLs we want. Right-click on an interesting object and select “Open in Object Store Browser.” That will direct to a login screen. After login, the UI brings up a page where it should display the query and give the output.</p> <center><img src="../../../static/images/basic_api_use_with_python/open_in_object_browser.jpg" /></center> <p>To see the URL it used we click on “Show URL and response of last query.”</p> <center><img src="../../../static/images/basic_api_use_with_python/See_url_from_browser.jpg" /></center> <p>You can navigate the API structure and find what you are looking for. It may take some practice to develop a full understanding of this method.</p> <p>I hope this has been helpful. I know it can be overwhelming, but the benefits of working with an API vs clicking though a web-page are wonderful. For example, before I left my last job I wrote a Python program that could be run on demand. It pulled what VLANS went to what VM Servers and looked for errors on interfaces. We got a call that there were lots of outages going on. The other network engineers were manually logging into everything looking for issues, but I just ran my program and was able to find a missing VLAN on a trunk to a VM server. The VM failed because it could no longer communicate out-side of the VM server. It took approximately five minutes to find and fix the issue. With dozens of VM servers and very little info on what the actual issue was, I believe that outage would have been much longer if people were looking for the issue by hand.</p> <p>-Daniel</p>Daniel HimesI come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.Using NTC Templates in Ansible2020-02-04T00:00:00+00:002020-02-04T00:00:00+00:00https://blog.networktocode.com/post/using_ntc_templates<p>Network to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created <a href="https://github.com/google/textfsm">TextFSM</a> that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a <code class="highlighter-rouge">show lldp neighbor</code> command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.</p> <p>You can find more about the <a href="https://github.com/networktocode/ntc-templates">ntc-templates on our Github page</a>. These are templates that have been created by both Network to Code and by the community. It is an open source project that anyone is able to contribute a template to if there is not a template already created. Take a look at the <a href="https://github.com/networktocode/ntc-templates">readme</a> for how you can help contribute to NTC Templates!</p> <h2 id="other-methods">Other Methods</h2> <p>Ansible is not the only method for using TextFSM to get structured data returned from network device output. Several Python libraries such as Netmiko and Nornir are able to use TextFSM to return structured data by setting a flag to use TextFSM as well.</p> <h2 id="using-ntc-templates-with-ansible">Using NTC-Templates with Ansible</h2> <p>There are two primary methods for sending data through a TextFSM parser in Ansible. You will get to see examples for both of these. The first method available is to use the TextFSM Filter. The second is in conjunction with the Ansible Galaxy role Network Engine.</p> <h3 id="getting-started">Getting Started</h3> <p>The first step towards leveraging these templates is getting the templates on to a compute device that is running Ansible. For this demo, we’ll be using a local machine and Git to clone the repository to a local directory.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:networktocode/ntc-templates.git </code></pre></div></div> <p>Next you will need to install TextFSM to the same Python interpreter that your Ansible installation is installed in. This demo will leverage Python3. For the installation you will also use the <code class="highlighter-rouge">--user</code> flag to install as the local user and <code class="highlighter-rouge">--upgrade</code> to ensure that the latest version of TextFSM is installed.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip3 <span class="nb">install </span>textfsm <span class="nt">--user</span> <span class="nt">--upgrade</span> </code></pre></div></div> <p>For the second method of using Ansible Galaxy’s Network Engine you will also need to install the role as follows:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-galaxy <span class="nb">install </span>ansible-network.network-engine </code></pre></div></div> <p>Now that the environment is setup it’s time to take a look at the Playbook and the execution.</p> <h3 id="playbook-setup-and-execution">Playbook Setup and Execution</h3> <p>This lab is setup as a 3 router traingle. RTR-1 will connect to one interface on RTR-2 and RTR-3. RTR-2 will have one connection to RTR-1 and RTR-3. RTR-3 will have the corresponding connections to RTR-1 and RTR-2.</p> <p><img src="../../../static/images/blog_posts/ansible-using-ntc-templates/using_templates_3router.png" alt="3_router_triangle" /></p> <h4 id="textfsm-cli-parser---ansible-built-in">TextFSM CLI Parser - Ansible Built In</h4> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">DEMO</span><span class="nv"> </span><span class="s">OF</span><span class="nv"> </span><span class="s">TEXTFSM"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">routers</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">COMMAND</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">show lldp neighbors</span> <span class="na">register</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">REGISTER</span><span class="nv"> </span><span class="s">OUTPUT</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">DEVICE_NEIGHBORS</span><span class="nv"> </span><span class="s">VARIABLE"</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">device_neighbors</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lldp_output.stdout[0]</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">parse_cli_textfsm('~/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm')</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">3:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_neighbors</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">4:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">NEIGHBORS"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}:</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item['NEIGHBOR']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_neighbors</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>Walking through this first play, in <em>Task 1</em> you are connecting to the device and running the command <code class="highlighter-rouge">show lldp neighbors</code>. This is saved to a variable named <code class="highlighter-rouge">lldp_output</code> that will be used later.</p> <p>In <em>Task 2</em> the output is being sent through the <strong>parse_cli_textfsm</strong> filter. The filter is being provided the library file to be used in the parsing. This is explicitely called out. The output from going through the TextFSM parser is then getting assigned to the variable <code class="highlighter-rouge">device_neighbors</code>, with each host in the execution having their own local instance of this variable.</p> <p>In <em>Task 3</em> you are getting to see the output of the variable. This shows the structured data. With this particular template you get the keys of <strong>CAPABILITIES</strong>, <strong>LOCAL_INTERFACES</strong>, <strong>NEIGHBOR</strong>, and <strong>NEIGHBOR_INTERFACE</strong> back. These are all defined within the TextFSM template.</p> <p>In <em>Task 4</em> you get to see a practical output to a screen if you want to audit and understand the neighbor relationships seen by LLDP on the network devices.</p> <p>In this example you need to set a fact to be able to access the data later or continue to send the output through the <code class="highlighter-rouge">parse_cli_textfsm</code> filter every time you want to get at the structured data, such as a line number 15 in TASK 2.</p> <h5 id="textfsm-cli-parser-output">TextFSM CLI Parser Output</h5> <p>Here is the output that corresponds with the play above.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-playbook demo_ansible_textfsm.yml PLAY <span class="o">[</span>PLAY 1: DEMO OF TEXTFSM WITH CLI PARSER] <span class="k">*************************************************************************</span> TASK <span class="o">[</span>TASK 1: GET COMMAND OUTPUT] <span class="k">**************************************************************************************</span> ok: <span class="o">[</span>rtr-2] ok: <span class="o">[</span>rtr-3] ok: <span class="o">[</span>rtr-1] TASK <span class="o">[</span>TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE] <span class="k">************************************************************</span> ok: <span class="o">[</span>rtr-1] ok: <span class="o">[</span>rtr-2] ok: <span class="o">[</span>rtr-3] TASK <span class="o">[</span>TASK 3: PRINT OUTPUT] <span class="k">********************************************************************************************</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/0"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-3"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/0"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-1"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/2"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-3"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-2"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-1"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/2"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/0"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-2"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/0"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> TASK <span class="o">[</span>TASK 4: PRINT NEIGHBORS FROM CLI PARSER] <span class="k">*************************************************************************</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/2<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/2: rtr-3"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-2"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/0<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/0: rtr-3"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-1"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-1"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/0<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/0: rtr-2"</span> <span class="o">}</span> PLAY RECAP <span class="k">*************************************************************************************************************</span> rtr-1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 rtr-2 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 rtr-3 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <h4 id="ansible-role---network-engine">Ansible Role - Network Engine</h4> <p>The Ansible Role - Network Engine is a role that extracts information about network devices into Ansible Facts. You can use either TextFSM syntax or YAML (<code class="highlighter-rouge">command_parser</code> option) to extract information about a network device from the command output. You can read more about using the role at the <a href="https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/README.md">network-engine Github page</a>.</p> <p>Here you will see a second method to get the same output with Network Engine instead of the CLI Parser. The biggest difference is that in this method, the Network Engine role will register the returned data to the ansible_facts instead of requiring you to set it to your own variable.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">DEMO</span><span class="nv"> </span><span class="s">OF</span><span class="nv"> </span><span class="s">TEXTFSM</span><span class="nv"> </span><span class="s">WITH</span><span class="nv"> </span><span class="s">NETWORK</span><span class="nv"> </span><span class="s">ENGINE"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">routers</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">roles</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">ansible-network.network-engine</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">COMMAND</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">show lldp neighbors</span> <span class="na">register</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">RUN</span><span class="nv"> </span><span class="s">THROUGH</span><span class="nv"> </span><span class="s">THE</span><span class="nv"> </span><span class="s">PARSER"</span> <span class="na">textfsm_parser</span><span class="pi">:</span> <span class="na">file</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/Users/ntcblog/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm"</span> <span class="na">content</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lldp_output.stdout[0]</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">name</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">3:</span><span class="nv"> </span><span class="s">SHOW</span><span class="nv"> </span><span class="s">ANSIBLE</span><span class="nv"> </span><span class="s">FACTS</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">4:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">NEIGHBORS</span><span class="nv"> </span><span class="s">FROM</span><span class="nv"> </span><span class="s">ANSIBLE</span><span class="nv"> </span><span class="s">NETWORK</span><span class="nv"> </span><span class="s">ENGINE"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}:</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item['NEIGHBOR']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['lldp_output']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>There is a new key that we needed to add to import the role that was installed with the <code class="highlighter-rouge">ansible-galaxy</code> command earlier. This is the <code class="highlighter-rouge">roles:</code> key that you see under <code class="highlighter-rouge">gather_facts</code> and contains a list of roles to import into the play. Here you see the import of the <code class="highlighter-rouge">ansible-network.network-engine</code> role.</p> <p>In <em>Task 1</em> you once again have the same command to gather the LLDP neighbors from the device.</p> <p>In <em>Task 2</em> instead of registering to a fact and sending through a filter, you now use the module <code class="highlighter-rouge">textfsm_parser</code> that takes <em>file</em> as a parameter that is the file of the TextFSM template. This is the same template referenced in the filter on the first play. You also pass <em>content</em> of what output you want to send through the parser. The last parameter is the <em>name</em> of the fact that you are registering to ansible_facts.</p> <p>In <em>Task 3</em>, you once again get to see the output for <code class="highlighter-rouge">ansible_facts</code>. This will show that there are more details available about the device.</p> <p>In <em>Task 4</em> you get to see the same output as <em>Play 1</em> where you get the neighbors printed out.</p> <h5 id="network-engine-output">Network Engine Output</h5> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">ansible-playbook demo_ansible_textfsm.yml</span> <span class="s">PLAY [PLAY 2</span><span class="pi">:</span> <span class="s">DEMO OF TEXTFSM WITH NETWORK ENGINE] *********************************************************************</span> <span class="s">TASK [TASK 1</span><span class="pi">:</span> <span class="s">GET COMMAND OUTPUT] **************************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">TASK [TASK 2</span><span class="pi">:</span> <span class="s">RUN THROUGH THE PARSER] **********************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">TASK [TASK 3</span><span class="pi">:</span> <span class="s">SHOW ANSIBLE FACTS OUTPUT] *******************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-3"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-3"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="s">TASK [TASK 4</span><span class="pi">:</span> <span class="s">PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE] *************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/2) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2:</span><span class="nv"> </span><span class="s">rtr-3"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-2"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/0) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0:</span><span class="nv"> </span><span class="s">rtr-3"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-1"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-1"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/0) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0:</span><span class="nv"> </span><span class="s">rtr-2"</span> <span class="err">}</span> <span class="s">PLAY RECAP *************************************************************************************************************</span> <span class="na">rtr-1 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> <span class="na">rtr-2 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> <span class="na">rtr-3 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> </code></pre></div></div> <h2 id="summary">Summary</h2> <p>This shows that there are multiple ways of pairing TextFSM templates with Ansible. Getting structured data out of unstructured output is extremely valuable when it comes to automating a network environment. There are over 300 different templates currently in the NTC-Templates repository to help get structured data out of your unstructured data.</p> <p>-Josh</p>Josh VanDeraaNetwork to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created TextFSM that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a show lldp neighbor command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.Versionable Database2020-01-28T00:00:00+00:002020-01-28T00:00:00+00:00https://blog.networktocode.com/post/Versionable-Database<p>One of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.</p> <h2 id="problem-space">Problem Space</h2> <p>When managing and storing your Source of Truth data, there are two primary mechanisms, each with it’s own pros and cons.</p> <p>Source Control (Git)</p> <ul> <li>Provides the ability to version data in such a way the exact state of the data at any point in the past is known.</li> <li>Provides the ability to know who is the owner of the data.</li> <li>Provides the ability to populate data in a staging area, without modifying the production data.</li> <li>Tooling integration with things such as CI Systems.</li> <li>Decentralized storage of the data.</li> </ul> <p>Database</p> <ul> <li>Provides <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>-based transactions of the data.</li> <li>Has a native querying language to obtain, filter, and all around work with the data.</li> <li>Provides schema enforcement of the data.</li> <li>The ability to scale to large datasets.</li> </ul> <p>There is a clear dichotomy between these two choices. On one hand there is great integrations with all standard DevOps tooling, on the other hand there is an enterprise-grade manager of the data.</p> <h2 id="history">History</h2> <p>For several years, a few of us have been searching for solutions that combine these two concepts. If one could build a database with Git constructs, this would allow versionable management of your data, with the ability to query, effectively store, and manage at scale. Through various searches it would seem that this has not been solved in any meaningful way. The closest I have come across was a project called <a href="https://github.com/attic-labs/noms">NOMS</a>, but it has limitations.</p> <p>Recently a startup company called <a href="https://www.liquidata.co/">liquidata</a> was founded and is creating a solution–a library called <a href="https://github.com/liquidata-inc/dolt">Dolt</a>. Dolt is is written on top of the NOMS project, in Go. The intention is to support all Git semantics (such as branch, add, commit, etc..), as well as all MySQL semantics (such as insert, create, update, etc…).</p> <p>By supporting all MySQL semantics, applications that use a MySQL server should be able to simply use a Dolt-SQL server with no disruption. Whether you are using an ODBC driver or raw SQL, in theory it should still work. You would naturally have to build in the capability for your application to take advantage of the Git capabilities or use traditional “Git-like” workflows.</p> <p>By supporting all Git semantics, data should be able to be managed in a decentralized fashion using Git workflows to branch, add, diff, and merge the data. In a similar vein to GitHub, they have developed DoltHub, to provide that level of tooling expected in a standard Git user interface, such as forking, API’s, webhooks, and CI integrations.</p> <h2 id="primer">Primer</h2> <p>In this brief introduction to the technology, we will create a Dolt data repository, called “simple-inventory” and run it on a dolt-sql server. If you care to follow along the only requirements are to have Docker and internet access.</p> <h2 id="setup">Setup</h2> <p>Start the Docker container.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="nt">--name</span> dolt golang:1.12.14 </code></pre></div></div> <blockquote> <p>Install Dolt by running <code class="highlighter-rouge">curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash</code></p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go# curl <span class="nt">-L</span> https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 601 0 601 0 0 2647 0 <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- 2647 100 3038 100 3038 0 0 8392 0 <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- 8392 Downloading: https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/dolt-linux-amd64.tar.gz Installing dolt, git-dolt and git-dolt-smudge to /usr/local/bin. root@2a29f8a34dd3:/go# </code></pre></div></div> <p>You can create a Dolt data repository by creating a folder, and then initializing the repository. Just like Git, in Dolt, you need to add yourself to the Dolt config. If you are familiar with Git, these commands will be familiar, only changing the command from <code class="highlighter-rouge">git</code> to <code class="highlighter-rouge">dolt</code>, but keeping all of the other options the same.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go# <span class="nb">mkdir </span>simple-inventory root@2a29f8a34dd3:/go# <span class="nb">cd </span>simple-inventory root@2a29f8a34dd3:/go/simple-inventory# dolt config <span class="nt">--global</span> <span class="nt">--add</span> user.email <span class="s2">"ken@celenza.org"</span> Config successfully updated. root@2a29f8a34dd3:/go/simple-inventory# dolt config <span class="nt">--global</span> <span class="nt">--add</span> user.name <span class="s2">"itdependsnetworks"</span> Config successfully updated. root@2a29f8a34dd3:/go/simple-inventory# dolt init Successfully initialized dolt data repository. root@2a29f8a34dd3:/go/simple-inventory# dolt status On branch master nothing to commit, working tree clean root@2a29f8a34dd3:/go/simple-inventory# dolt log commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <h2 id="creating-tables">Creating Tables</h2> <p>Ideally, if you are familiar with MySQL the only difference should be using Dolt engine instead of a MySQL server, everything should be the same. To enter the Dolt SQL server, simply use the command <code class="highlighter-rouge">dolt sql</code>.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <p>Create tables as you normally would, in this example, we will create two tables. The table <code class="highlighter-rouge">device_inventory</code> will have the inventory of the devices, with columns for hostname and IP address. The table <code class="highlighter-rouge">vlan</code> will have the hostname (with foreign key relationaship to the <code class="highlighter-rouge">device_inventory</code> table) column, VLAN , and name of the VLAN.</p> <blockquote> <p>Note: Though the foreign key relationaship syntax is defined, it is not a currently supported Dolt feature.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">device_inventory</span> <span class="p">(</span> <span class="o">-&gt;</span> <span class="n">hostname</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="n">ip_address</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="k">primary</span> <span class="k">key</span> <span class="p">(</span><span class="nv">`hostname`</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">);</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">vlan</span> <span class="p">(</span> <span class="o">-&gt;</span> <span class="n">hostname</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">),</span> <span class="o">-&gt;</span> <span class="n">vlan</span> <span class="nb">int</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="n">name</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">),</span> <span class="o">-&gt;</span> <span class="k">FOREIGN</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">hostname</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">device_inventory</span><span class="p">(</span><span class="n">hostname</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">);</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> </code></pre></div></div> <p>Here is where it starts to get interesting–we can combine these two different concepts and see how they interact. So far, the data has not been committed into master, only staged into our local environment. We can view this by issuing standard Git-like commands when we return back to the command line from the dolt-sql server.</p> <blockquote> <p>Run the <code class="highlighter-rouge">dolt status</code>, <code class="highlighter-rouge">dolt diff</code>, and <code class="highlighter-rouge">dolt diff -q</code> commands.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt status On branch master Untracked files: <span class="o">(</span>use <span class="s2">"dolt add &lt;table&gt;"</span> to include <span class="k">in </span>what will be committed<span class="o">)</span> new table: device_inventory new table: vlan root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/device_inventory b/device_inventory added table diff <span class="nt">--dolt</span> a/vlan b/vlan added table root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> CREATE TABLE <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span> <span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:0'</span>, <span class="sb">`</span>ip_address<span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:1'</span>, PRIMARY KEY <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">)</span> <span class="o">)</span><span class="p">;</span> CREATE TABLE <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span> <span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:0'</span>, <span class="sb">`</span>vlan<span class="sb">`</span> BIGINT NOT NULL COMMENT <span class="s1">'tag:1'</span>, <span class="sb">`</span>name<span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:2'</span>, PRIMARY KEY <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span><span class="o">)</span> <span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>The <code class="highlighter-rouge">dolt status</code> command shows that the schema has not “taken effect” by being merged into master. The <code class="highlighter-rouge">dolt diff</code> with optional <code class="highlighter-rouge">-q</code> flag, shows that a new table has been created. Note the additional <code class="highlighter-rouge">tag</code> parameters, this is logic to track column names, which may change over time and cause name conflicts. The new schema can be committed into master.</p> <blockquote> <p>Issue the commands <code class="highlighter-rouge">dolt add -a</code>, <code class="highlighter-rouge">dolt commit -m 'initial schema'</code>, and <code class="highlighter-rouge">dolt log</code>.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'initial schema'</span> commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema root@2a29f8a34dd3:/go/simple-inventory# dolt log commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>Awesome! The first piece of information of the database was committed.</p> <h2 id="adding-data">Adding data</h2> <p>Defining schema is not all the valuable without the data. Again, using standard SQL constructs we can commit the data. Let’s use proper branching strategies this time.</p> <blockquote> <p>Create a branch and Enter dolt-sql</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout <span class="nt">-b</span> intial_data Switched to branch <span class="s1">'intial_data'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>Add data</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">device_inventory</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">ip_address</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="nv">"10.1.1.1"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">device_inventory</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">ip_address</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="nv">"10.1.1.2"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">"user"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="nv">"printer"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="nv">"wap"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">"user"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="nv">"printer"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="nv">"wap"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> <span class="n">Bye</span> </code></pre></div></div> <p>Now that we have staged the data, we can view the date in two different ways. The first is via raw sql entries, and the other is more akin to a unix diff.</p> <blockquote> <p>View the diff. In the console, diffs are actually color coded green and red for add and remove respectively.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/device_inventory b/device_inventory <span class="nt">---</span> a/device_inventory @ tktob1spsoos6isfdj4o9benp9c57iic +++ b/device_inventory @ b37haqr0n9j1e5bckj35sqt4kntojlp9 +-----+----------+------------+ | | <span class="nb">hostname</span> | ip_address | +-----+----------+------------+ | + | nyc-sw01 | 10.1.1.1 | | + | nyc-sw02 | 10.1.1.2 | +-----+----------+------------+ diff <span class="nt">--dolt</span> a/vlan b/vlan <span class="nt">---</span> a/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn +++ b/vlan @ e2kovelv2sfjuo584o0v6hp76207k720 +-----+----------+------+---------+ | | <span class="nb">hostname</span> | vlan | name | +-----+----------+------+---------+ | + | nyc-sw01 | 10 | user | | + | nyc-sw01 | 20 | printer | | + | nyc-sw01 | 30 | wap | | + | nyc-sw02 | 10 | user | | + | nyc-sw02 | 20 | printer | | + | nyc-sw02 | 30 | wap | +-----+----------+------+---------+ root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> INSERT INTO <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>ip_address<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,<span class="s2">"10.1.1.1"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>ip_address<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,<span class="s2">"10.1.1.2"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,10,<span class="s2">"user"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,20,<span class="s2">"printer"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,30,<span class="s2">"wap"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,10,<span class="s2">"user"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,20,<span class="s2">"printer"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,30,<span class="s2">"wap"</span><span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>Personally, I’m pretty impressed with what is happening here, but there is still more to do to complete this workflow. The data needs to be committed, and merged from the “feature” branch into master branch.</p> <blockquote> <p>Commit to the feature branch.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'initial data'</span> commit gbt6r21mtbd0cvs5iahog912essjhn0n Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:50 +0000 2020 initial data root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <blockquote> <p>Merge into the master branch.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master Switched to branch <span class="s1">'master'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt branch intial_data <span class="k">*</span> master root@2a29f8a34dd3:/go/simple-inventory# dolt merge intial_data Updating 4n29eaqb1v9pmnpnda2r43fc91p1vp9l..gbt6r21mtbd0cvs5iahog912essjhn0n Fast-forward root@2a29f8a34dd3:/go/simple-inventory# dolt log commit gbt6r21mtbd0cvs5iahog912essjhn0n Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:50 +0000 2020 initial data commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>So far this is showing the “Create” part of standard CRUD operations is working, meaning we can add data to the repository.</p> <h2 id="update-data">Update Data</h2> <p>Naturally, as soon as data is entered, you will want to modify it. The process is the same to modify: branch, sql statements, add, commit, and merge.</p> <blockquote> <p>Checkout a new branch, and enter Dolt SQL.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout <span class="nt">-b</span> change_vlan Switched to branch <span class="s1">'change_vlan'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>Update the data.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">update</span> <span class="n">vlan</span> <span class="k">set</span> <span class="n">name</span> <span class="o">=</span> <span class="nv">"prnt"</span> <span class="k">where</span> <span class="n">name</span> <span class="o">=</span> <span class="nv">"printer"</span><span class="p">;</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="o">|</span> <span class="n">matched</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> <span class="n">Bye</span> </code></pre></div></div> <blockquote> <p>View the diff.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/vlan b/vlan <span class="nt">---</span> a/vlan @ bjg5ou111lmu03ciu6aisphf7m6jbk1r +++ b/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn +-----+----------+------+---------+ | | <span class="nb">hostname</span> | vlan | name | +-----+----------+------+---------+ | &lt; | nyc-sw01 | 20 | printer | | <span class="o">&gt;</span> | nyc-sw01 | 20 | prnt | | &lt; | nyc-sw02 | 20 | printer | | <span class="o">&gt;</span> | nyc-sw02 | 20 | prnt | +-----+----------+------+---------+ root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> UPDATE <span class="sb">`</span>vlan<span class="sb">`</span> SET <span class="sb">`</span>name<span class="sb">`</span><span class="o">=</span><span class="s2">"prnt"</span> WHERE <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">=</span><span class="s2">"nyc-sw01"</span> AND <span class="sb">`</span>vlan<span class="sb">`</span><span class="o">=</span>20<span class="o">)</span><span class="p">;</span> UPDATE <span class="sb">`</span>vlan<span class="sb">`</span> SET <span class="sb">`</span>name<span class="sb">`</span><span class="o">=</span><span class="s2">"prnt"</span> WHERE <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">=</span><span class="s2">"nyc-sw02"</span> AND <span class="sb">`</span>vlan<span class="sb">`</span><span class="o">=</span>20<span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>A keen eye will note the captured update statements shown in the diff are not the same as the update statement inputted in the SQL server. This conversion makes sense, since the data could be merged on a different set of data that had more or less data in it. It also illustrates the complexity of building this technology.</p> <blockquote> <p>Commit to a feature branch and merge to master.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'update data'</span> commit jqe3vb84fhfunn7uapksdt5thb4rr7r7 Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:23:03 +0000 2020 update data root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master Switched to branch <span class="s1">'master'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt merge change_vlan Updating gbt6r21mtbd0cvs5iahog912essjhn0n..jqe3vb84fhfunn7uapksdt5thb4rr7r7 Fast-forward root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>View the data from dolt sql.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">vlan</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="o">|</span> <span class="n">hostname</span> <span class="o">|</span> <span class="n">vlan</span> <span class="o">|</span> <span class="n">name</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="k">user</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">20</span> <span class="o">|</span> <span class="n">prnt</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">30</span> <span class="o">|</span> <span class="n">wap</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="k">user</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">20</span> <span class="o">|</span> <span class="n">prnt</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">30</span> <span class="o">|</span> <span class="n">wap</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> </code></pre></div></div> <p>You can also view who is the owner of the data by tracking to the commit, which includes the auther, time, and commit hash metadata as well, using the <code class="highlighter-rouge">dolt blame &lt;table&gt;</code> command.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt blame vlan +----------+------+--------------+-------------------+------------------------------+-------------------+ | HOSTNAME | VLAN | COMMIT MSG | AUTHOR | TIME | COMMIT | +----------+------+--------------+-------------------+------------------------------+-------------------+ | nyc-sw01 | 30 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw02 | 10 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw02 | 20 | update data | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua | | nyc-sw02 | 30 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw01 | 10 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw01 | 20 | update data | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua | +----------+------+--------------+-------------------+------------------------------+-------------------+ root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <h2 id="summary">Summary</h2> <p>This is just a primer, and there is still a lot of work to be done to truly have feature parity with both MySQL and Git, but this is a great step in the right direction. I plan to continue to monitor and follow up with examples, use cases, and library updates over time. Specifically, next time, I want to extend the workflow to include DoltHub to review those capabilities as well.</p> <p>-Ken</p>Ken CelenzaOne of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.Ansible gather_facts for Networking Devices2020-01-24T00:00:00+00:002020-01-24T00:00:00+00:00https://blog.networktocode.com/post/ansible-gather_facts-enabled<p>The purpose of this blog is to show some differences in how Ansible handled gathering facts from networking devices prior to version 2.9 and how they can be gathered in version 2.9.2. A main point to show here is how previously the <code class="highlighter-rouge">gather_facts</code> key was disabled when working with networking devices and now it can be enabled to gather device facts.</p> <h2 id="gathering-_facts-the-old-way">Gathering *_facts the “Old” way</h2> <p>Ansible core comes included with vendor specific <code class="highlighter-rouge">*_facts</code> modules. So normally to collect facts from devices the playbook was built using the vendor specific facts module and the needed key was used for any other particular task like asserting device versions, building reports, etc. The example below is a playbook used to gather facts from a Cisco device.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR IOS</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">csr1</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="c1">#&lt;---Gather Facts Disabled</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR IOS IN A TASK</span> <span class="na">ios_facts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW ALL ANSIBLE FACT KEYS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_facts.keys()</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW HOSTNAME</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_hostname</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW OS VERSION</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_version</span> </code></pre></div></div> <blockquote> <p>Note: The first task is using the <code class="highlighter-rouge">ios_facts</code> module to gather all the data, the other three tasks are used to display the data returned from the facts module and keys inside <code class="highlighter-rouge">ansible_facts</code> can be accessed directly.</p> </blockquote> <p>Notice how in the play definition the key <code class="highlighter-rouge">gather_facts</code> is set to <strong>no</strong>. Originally, Ansible was designed to work with Linux servers. Ansible focuses on collecting as much data as possible from the remote Linux servers and copying Python modules to the servers for automation tasks so they could be executed on the remote machines.</p> <p><img src="../../../static/images/blog_posts/linux_servers.png" alt="" /></p> <p>When working with networking devices Ansible does <strong>not</strong> copy Python modules to the remote devices to run an automation task. The Python module is executed in the local Ansible workstation and things like sending show commands or configuration changes are wrapped in the Python module and sent over through SSH. The <code class="highlighter-rouge">gather_facts</code> key was not designed to know the difference between a Linux server and a networking device, so by enabling the feature would end up gathering facts from the local Ansible workstation and not the remote networking device.</p> <p><img src="../../../static/images/blog_posts/networking.png" alt="" /></p> <h2 id="running-the-playbook-with-gather_facts-disabled">Running the Playbook with gather_facts disabled</h2> <p>The output of the current playbook would look like this:</p> <blockquote> <p>Note: The current Ansible version is 2.7.10</p> </blockquote> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ntc@jump-host:~<span class="nv">$ </span>ansible-playbook <span class="nt">-i</span> inventory network_facts.yml PLAY <span class="o">[</span>GATHER FACTS FOR IOS] <span class="k">********************************************************************************</span> TASK <span class="o">[</span>GATHER FACTS FOR IOS] <span class="k">********************************************************************************</span> ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>VIEW ALL ANSIBLE FACT KEYS] <span class="k">**************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_facts.keys()"</span>: <span class="s2">"dict_keys(['net_serialnum', 'net_all_ipv4_addresses', 'net_model', 'net_hostname', 'net_gather_subset', 'net_filesystems_info', 'net_interfaces', 'net_version', 'net_neighbors', 'net_all_ipv6_add resses', 'net_memtotal_mb', 'net_filesystems', 'net_image', 'net_memfree_mb'])"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW HOSTNAME] <span class="k">***************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_hostname"</span>: <span class="s2">"csr1"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW OS VERSION] <span class="k">*************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_version"</span>: <span class="s2">"16.08.01a"</span> <span class="o">}</span> PLAY RECAP <span class="k">*************************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 </code></pre></div></div> <p>The current output only displays facts that belong to the remote device. The keys seen in the output are the ones available from the <code class="highlighter-rouge">ios_facts</code> module.</p> <h2 id="running-the-playbook-with-gather_facts-enabled">Running the Playbook with gather_facts enabled</h2> <p>When the <code class="highlighter-rouge">gather_facts</code> key is enabled in the play definition (<code class="highlighter-rouge">gather_facts: yes</code>) the output will return not only data from the remote device but also from the Ansible workstation as seen below:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ntc@jump-host:~<span class="nv">$ </span>ansible-playbook <span class="nt">-i</span> inventory network_facts.yml PLAY <span class="o">[</span>GATHER FACTS FOR IOS] <span class="k">********************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">*************************************************************************************</span> ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>GATHER FACTS FOR IOS] <span class="k">********************************************************************************</span> ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>VIEW ALL ANSIBLE FACT KEYS] <span class="k">**************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_facts.keys()"</span>: <span class="s2">"dict_keys(['module_setup', 'distribution_version', 'distribution_file_variety', 'env', 'userspace_bits', 'architecture', 'default_ipv4', 'swapfree_mb', 'default_ipv6', 'cmdline', 'selinux', 'userspace_architecture', 'product_uuid', 'pkg_mgr', 'distribution', 'iscsi_iqn', 'all_ipv6_addresses', 'uptime_seconds', 'kernel', 'system_capabilities_enforced', 'python', 'is_chroot', 'user_shell', 'product_serial' 'form_factor', 'distribution_file_parsed', 'fips', 'user_id', 'selinux_python_present', 'ansible_local', 'processor_vcpus', 'processor', 'ssh_host_key_ecdsa_public', 'mounts', 'system_vendor', 'swaptotal_mb', 'distribution_major_version', 'real_group_id', 'lsb', 'machine', 'ssh_host_key_rsa_public', 'user_gecos', 'processor_threads_per_core', 'eth1', 'product_name', 'all_ipv4_addresses', 'python_version', 'product_version', 'service_mgr', 'memory_mb', 'user_dir', 'gather_subset', 'real_user_id', 'virtualization_role', 'tunl0', 'dns', 'effective_group_id', 'lo', 'memtotal_mb', 'device_links', 'apparmor', 'memfree_mb', 'processor_count', 'hostname', 'interfaces', 'machine_id', 'fqdn', 'user_gid', 'nodename', 'domain', 'distribution_file_path', 'virtualization_type', 'ssh_host_key_ed25519_public', 'processor_cores', 'bios_version', 'date_time', 'distribution_release', 'os_family', 'effective_user_id', 'sit0', 'system', 'devices', 'user_uid', 'ssh_host_key_dsa_public', 'bios_date', 'system_capabilities', 'ens3', 'net_serialnum', 'net_all_ipv4_addresses', 'net_model', 'net_hostname', 'net_gather_subset', 'net_filesystems_info', 'net_interfaces', 'net_version', 'net_neighbors', 'net_all_ipv6_addresses', 'net_memtotal_mb', 'net_filesystems', 'net_image', 'net_memfree_mb'])"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW HOSTNAME] <span class="k">***************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_hostname"</span>: <span class="s2">"csr1"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW OS VERSION] <span class="k">*************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_version"</span>: <span class="s2">"16.08.01a"</span> <span class="o">}</span> PLAY RECAP <span class="k">*************************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>5 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 </code></pre></div></div> <h2 id="gathering-facts-from-different-platforms">Gathering Facts from different platforms</h2> <p>Another thing to point out is that when using the core <code class="highlighter-rouge">*_facts</code> modules if another vendor or platform needs to be added to the playbook another <strong>play</strong> would need to be built to gather facts from that vendor or platform. Without this addition, the module will fail because the <code class="highlighter-rouge">ios_facts</code> module is vendor specific and it will try to gather facts from any devices targeted in the scope of that play. The playbook would look something like this:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR IOS</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">csr1</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR IOS</span> <span class="na">ios_facts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW ALL ANSIBLE FACT KEYS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_facts.keys()</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW HOSTNAME</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_hostname</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW OS VERSION</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_version</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR NXOS</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">nxos</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS FOR NXOS</span> <span class="na">nxos_facts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW ALL ANSIBLE FACT KEYS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_facts.keys()</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW HOSTNAME</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_hostname</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW OS VERSION</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_version</span> </code></pre></div></div> <h2 id="gather-facts-the-new-way">Gather Facts the “New” way</h2> <p>There have been a lot of changes with Ansible focused on improving how to manage networking devices. More and more the platform has been used in multi-vendor environments, meaning that the tool needs to be more robust and vendor neutral. Recent features and improvements are being built to be more “vendor neutral” - such as <code class="highlighter-rouge">cli_config</code> and <code class="highlighter-rouge">cli_command</code> (added in Ansilble version 2.7).</p> <p>In Ansible version 2.9, gathering facts from devices from the play definition is now possible without having to build it as a task, like shown in the previous playbook. This time the key <code class="highlighter-rouge">gather_facts</code> can be enabled and not gather facts from the local workstation but from the devices targeted in the <code class="highlighter-rouge">hosts</code> key.</p> <blockquote> <p>Note: The modules <code class="highlighter-rouge">ios_facts</code> and <code class="highlighter-rouge">nxos_fact</code> are still being used but are now being executed in the play definition.</p> </blockquote> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GATHER FACTS</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">csr1,nxos</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">yes</span> <span class="c1">#&lt;----Gather Facts Enabled </span> <span class="na">tasks</span><span class="pi">:</span> <span class="c1">#NO VENDOR SPECIFIC MODULE LIKE ios_facts OR nxos_facts ANYMORE</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW ALL ANSIBLE FACT KEYS</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_facts.keys()</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW HOSTNAME</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_hostname</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VIEW OS VERSION</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">ansible_net_version</span> </code></pre></div></div> <p>Take a look at the current playbook. The <code class="highlighter-rouge">hosts</code> key has two different operating systems which normally would require another play when trying to gather data from different platforms because of the vendor specific modules defined in the tasks. This time gathering facts is being done at the play definition level and at the tasks level only the <code class="highlighter-rouge">debug</code> modules are being used to display the available keys and variables to see hostname and version information.</p> <h2 id="running-the-playbook-with-gather_facts-enabled-the-new-way">Running the Playbook with gather_facts enabled the “New Way”</h2> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ntc@jump-host:~<span class="nv">$ </span>ansible-playbook <span class="nt">-i</span> inventory network_facts.yml PLAY <span class="o">[</span>GATHER FACTS] <span class="k">***************************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">************************************************************************************</span> ok: <span class="o">[</span>csr1] ok: <span class="o">[</span>nxos] TASK <span class="o">[</span>VIEW ALL ANSIBLE FACT KEYS] <span class="k">*************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_facts.keys()"</span>: <span class="s2">"dict_keys(['network_resources', 'net_gather_network_resources', 'net_gather_subset', 'net_system', 'net_model', 'net_image', 'net_version', 'net_hostname', 'net_api', 'net_python_version', 'net_iostype', 'net_serialnum', 'net_filesystems', 'net_filesystems_info', 'net_memtotal_mb', 'net_memfree_mb', 'net_config', 'net_all_ipv4_addresses', 'net_all_ipv6_addresses', 'net_neighbors', 'net_interfaces', '_facts_gathered'])"</span> <span class="o">}</span> ok: <span class="o">[</span>nxos] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_facts.keys()"</span>: <span class="s2">"dict_keys(['network_resources', 'net_gather_network_resources', 'net_gather_subset', 'net__os', 'net__platform', 'net__hostname', 'net_interfaces_list', 'net_vlan_list', 'net_module', 'net_fan_info', 'net_power_supply_info', 'net_filesystems', 'net_memtotal_mb', 'net_memfree_mb', 'net_features_enabled', 'net_all_ipv4_addresses', 'net_all_ipv6_addresses', 'net_neighbors', 'net_interfaces', 'net_serialnum', 'net_license_hostid', 'net_system', 'net_model', 'net_image', 'net_version', 'net_platform', 'net_hostname', 'net_api', 'net_python_version', 'net_config', '_facts_gathered'])"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW HOSTNAME] <span class="k">***************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_hostname"</span>: <span class="s2">"csr1"</span> <span class="o">}</span> ok: <span class="o">[</span>nxos] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_hostname"</span>: <span class="s2">"nxos-spine1"</span> <span class="o">}</span> TASK <span class="o">[</span>VIEW OS VERSION] <span class="k">**************************************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_version"</span>: <span class="s2">"16.08.01a"</span> <span class="o">}</span> ok: <span class="o">[</span>nxos] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"ansible_net_version"</span>: <span class="s2">"7.0(3)I7(4)"</span> <span class="o">}</span> PLAY RECAP <span class="k">*************************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 nxos : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>The output above shows the results of the playbook ran in the new version. This time only the device facts are gathered in a single play for two different platforms, rather than running vendor specific modules that have to run in two different plays.</p> <blockquote> <p>Note: Now that <code class="highlighter-rouge">gather_facts</code> can be used with networking devices there are some configurations that can be changed in the <code class="highlighter-rouge">ansible.cfg</code> file that will affect how the gathering could behave.</p> </blockquote> <p>Changing the settings below in the <code class="highlighter-rouge">ansible.cfg</code> file will affect how gathering facts from devices will be gathered.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># smart - gather by default, but don't regather if already gathered</span> <span class="c"># implicit - gather by default, turn off with gather_facts: False</span> <span class="c"># explicit - do not gather by default, must say gather_facts: True</span> <span class="c"># gathering = implicit</span> </code></pre></div></div> <p>For more information on what other new changes exist in Ansible 2.9 check out the link below:</p> <p><a href="https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.9.html#id1">Ansible 2.9 Porting Guide</a></p> <h3 id="summary">Summary</h3> <p>Before this new update in Ansible 2.9.2, when building playbooks for networking devices the <code class="highlighter-rouge">gather_facts</code> key was always disabled and it almost seemed like an extra key that needed to be there for no good reason unless facts from the local machine needed to be added. In this new version of Ansible gathering facts from networking devices really makes things easier and more flexible in a structured way. I’m looking forward to test this out with the new <a href="https://www.ansible.com/deep-dive-into-ansible-network-resource-module">rersource modules</a> and showing this feature to more of my students from now on.</p> <p>-Hector</p>Hector IsazaThe purpose of this blog is to show some differences in how Ansible handled gathering facts from networking devices prior to version 2.9 and how they can be gathered in version 2.9.2. A main point to show here is how previously the gather_facts key was disabled when working with networking devices and now it can be enabled to gather device facts.Workflow and Productivity Management2020-01-14T00:00:00+00:002020-01-14T00:00:00+00:00https://blog.networktocode.com/post/Workflow-and-Productivity-Management<p>Full disclosure before you read any further: I am one of <em>“those”</em> people …the productivity obsessed. I am constantly trying to improve my output and overall workflow - both professionally and personally. I am by no means the most productive person in the world, but I’m constantly trying to work toward some unatainable level of expert efficiency. I’ve read <a href="https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People">7 Habits of Highly Effective People</a> and <a href="https://jamesclear.com/atomic-habits">Atomic Habits</a> (which I HIGHLY recommend). I’ve read every Lifehacker “How I Work” post. I’ve gone from vim to emacs back to vim - currently with highly-customized <a href="https://github.com/dirtyonekanobi/dotfiles/blob/master/vim/vimrc">vimrc</a> file. And, I’m currently in the process of trying to reset my sleep schedule to wake up at 4:00am, to reach the pinnacle of productivity!</p> <p>Take everything below with a grain of salt - there are probably much more scientific and accurate explanations for the information I provide herein. But, to quote the prophet Adam Sandler in his critically acclaimed philosophical sonnet “Wedding Singer”:</p> <blockquote> <p>I am the one with the Microphone! <img src="../../../static/images/blog_posts/workflow_productivity/wedding_singer.png" alt="" /></p> </blockquote> <p>As network engineers, developers, devops practictioners, and overall technologists, we have become accustomed to working in teams. These teams are often organized into functional or cross-functional groups that work together to accomplish a certain set of tasks - deliver an application; design, implement, maintain, augment the infrastructure that supports an application; rinse &amp; repeat. As such, there are many methodologies that define how teams work together to accomplish their set of tasks. A non-exhaustive list would include:</p> <ul> <li>Agile</li> <li>Kanban</li> <li>Waterfall</li> <li>Scrum (which, I think is Agile?)</li> <li>Lean</li> <li>Extreme Programming</li> <li>GTD</li> <li>And many…many…more (so many more)</li> </ul> <p>These methodologies, for better or worse, define how a <strong>project</strong> is managed. They establish a set of standards and working cadence that facilitates a few expected outcomes: delivery, collaboration, acceptance criteria, estimation, and scheduling. When utilized correctly, these methodolgies can be powerful tools to manage a team toward an efficent and euphoric high-functioning juggernaut!</p> <p>The question that can often get overlooked, even on these high-functioning championship teams is: How do I manage <strong>MY</strong> time? How do <strong>I</strong> prioritize and maximize <strong>MY</strong> productivity?</p> <p>With that question in mind, we will take a look at a method of personal productivity management that can be beneficial for anyone working in an engineering/application/devops capacity.</p> <h2 id="priority-vs-productivity">Priority vs Productivity</h2> <p>In all my research, tweaks, failures, and successes, I’ve identified that there is a big distinction to be made between priority and productivity. They work hand in hand, and are equally important toward an efficient workflow. <strong>Priorty</strong> identifies what to do at a given moment; <strong>Productivity</strong> is how you do it. Being proficient at both is the key to maximizing our time and potential as engineers.</p> <p>More often that not, external factors have much influence on prioritization. Deadlines, dependencies, budget contraints, outages, threats, public shamings, etc… can all impact which tasks get pulled into our “hot list”. Often times this is out of our control.</p> <p>Prioritization can be summed up by using the common computer science analogy of the <a href="https://gribblelab.org/CBootCamp/7_Memory_Stack_vs_Heap.html">Stack vs Heap</a>.</p> <p>In this analogy, “the Stack” involves things that are done quickly, with very little congitive load. This could involve answering an email, documenting a new feature, commenting on a pull-request, reviewing a pull-request (sometimes), and other “quick” tasks. The key here is the cognitive load. A coworker of mine is an all-time champ at answering questions on the Network to Code Slack channel. During downtime he can engage with someone, diagnose their issue, and respond, almost asyncronously from his other tasks. He’ll do this if he has 5 minutes between meetings, waiting on a response from one of his employees, etc… He is able to do this based on experience and muscle memory - most questions require very little research for him. If a question does require additional research or investigation, he may refrain from responding or file it away for a later time …when he has time for “the Heap”.</p> <p>“The Heap” in computer science refers to operations requiring memory allocations of a non-finite size. As such, it is more flexible, but more expensive than operations involving “the Stack”. As technologists, examples of using “The Heap” could include: reviwing documentation, writing code, debugging, troubleshooting, researching a new topic, reviewing a design request, and code review.</p> <p>The Stack vs Heap analogy highlights the importance of properly identifying a task or request’s scope, and what is involved in fulfilling/resolving it. A good question to ask is: “Will this take me more than 5 minutes?” If so, it is normally left for an allocated time of focus, where the task can be given the attention it needs. Otherwise, it can be handled immediately, or during a time requiring less focus. Some answer emails and handle other “quick” tasks at pre-allocated times during the day (more on this later); others handle these tasks as they come in, but only for a short period of time, before returning to a more involved task (this is often the case for those in an operations role).</p> <h3 id="proper-prioritization-prevents-poor-performance">Proper Prioritization Prevents Poor Performance</h3> <p>The biggest takeaway from the Productivity vs Priority distinction is that it is imperitive that we identify our priorities and make a plan to execute on them. Conversely, it is important that we identify secondary responsiblities and potential distractions that can negatively impact our priorities.</p> <p>If our main job responsibility is to deliver a feature by the end of a sprint, will our team care how many emails we have answered if the release is late? If we are asked to resolve tickets quickly to improve our team’s MTTR, does researching new features contribute to, or detract from that goal?</p> <p>Though those questions may sound bleak, the good news is that we normally have more time for productivity that we think. This is especially true if we work in an environment with reasonable deadlines and efficient estimation - e.g., Network to Code (<em>shameless plug</em>). Normally, we have time to deliver on our primary, and additional responsiblities and look for ways to improve efficiency - if we improve our productivity.</p> <h2 id="pomodoro-technique---embrace-the-tomato">Pomodoro Technique - Embrace the Tomato</h2> <p>The method I have personally found to be most productive <em><strong>FOR MYSELF</strong></em> is called <a href="https://en.wikipedia.org/wiki/Pomodoro_Technique">“Pomodoro”</a>. I have found great success utilizing this method because it forces you to allocate both focused and unfocused time, along with breaks. In short, the Pomodoro Technique consists of the following:</p> <ul> <li>A “Pomodoro” is a CONSECUTIVE unit of focused time - normally 25 minutes</li> <li>Each pomodoro is followed by a short (3-5 minute) or long (15-30 minute) break</li> <li>Breaks increase in length as successful pomodoros are completed</li> <li>Tasks are handled in a First In First Out method</li> <li>If a tasks takes longer than 25 minutes, it can be completed after a break during the next pomodoro</li> <li>If a task takes less than 25 minutes the remaining time can be left for unfocused (“Stack”) activities</li> <li>WRITE EVERYTHING DOWN!! Take notes on tasks, successful/failed pomodoros, timing, etc…</li> </ul> <p>It also promotes good prioritization, planning, estimation, and self-reflection. At the end of a day, week, or sprint, it is easy to identify where time was spent, and what optimizations are needed.</p> <p>Here are some things I’ve done to utilize the pomodoro technique effectively:</p> <ul> <li>Assign every task a due date as you receive it</li> <li>At the beginning of the week define the week’s goals and the total time you’re allocating to work (time available minus meetings, appts, personal time, PTO)</li> <li>At the beginning of the day write down Things To Do, keeping in mind the time available for that day</li> <li>Identify which tasks are most difficult, and/or least desireable, and stagger them with more desireable/easy tasks</li> <li>Always start with a desireable/easy task to build momentum</li> <li>Schedule long breaks, lunches, and personal time, by adding these things to your calendar</li> <li>Mute Slack, email, and other notifications (if possible). Move your phone to another room, or away from your reach</li> <li>Have fun &amp; execute</li> </ul> <p>With all of these productivity “tips”, it is important to remember that “to err is human”. Some days I am a productivity machine. Some days I can barely type my own name. However, I can personally attest to the success of the pomodoro method. When I use it effectively, at the end of each day, week, sprint, my output is optimal, and I feel great about it. When everything is “on fire”, time seems to fly by, with little impact.</p> <h2 id="in-conclusion-just-pick-something">In Conclusion …Just Pick Something</h2> <p>Is the Pomodoro Technique the “silver bullet” for productivity? Surely - just like Agile, DevOps, and Kubernetes solve everything. This technique may not be the best for you. But, the principles of Prioritization and Productivity can help to contribute to an improved workflow, and hopefully make us all more effective and efficient. The most important thing is to be more deliberate with our time, and to pick a technique, method, and workflow, that works for each of us individually. Increasing productivity is the closest we can get to building a time machine, in that it can help us waste less of it. Maybe that justifies my obsession?</p> <p>-Daryn</p>Daryn JohnsonFull disclosure before you read any further: I am one of “those” people …the productivity obsessed. I am constantly trying to improve my output and overall workflow - both professionally and personally. I am by no means the most productive person in the world, but I’m constantly trying to work toward some unatainable level of expert efficiency. I’ve read 7 Habits of Highly Effective People and Atomic Habits (which I HIGHLY recommend). I’ve read every Lifehacker “How I Work” post. I’ve gone from vim to emacs back to vim - currently with highly-customized vimrc file. And, I’m currently in the process of trying to reset my sleep schedule to wake up at 4:00am, to reach the pinnacle of productivity!Automation Principles - DRY2020-01-07T00:00:00+00:002020-01-07T00:00:00+00:00https://blog.networktocode.com/post/Principle-Series-DRY<p>This is part of a <a href="../Network-Automation-Principles/">series of posts</a> focused on Network Automation Principles.</p> <h2 id="dry-in-computer-science">DRY in Computer Science</h2> <p>The phrase <code class="highlighter-rouge">Do Not Repeat Yourself</code> or <code class="highlighter-rouge">DRY</code> is a rather simple concept. If you are writing a piece of code multiple times, modularize the code in a way to ensure you are not copying code multiple times. The definition of DRY states:</p> <blockquote> <p>“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”.</p> </blockquote> <p>Once there are multiple copies, inevitably, one of the pieces of code will not function the same as others, based on some missed step. Additionally if you want to change functionality in one place, you would have to remember to change it in all places.</p> <p>To put in context, Bart Simpson writing the same phrase on the chalkboard over and over is certainly not DRY.</p> <p><img src="../../../static/images/blog_posts/dry/simpsons.png" alt="Bart becoming DRY" /></p> <h2 id="example">Example</h2> <p>In this first contrived example you will see the duplication of code in how the VLAN data is being validated.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">add_vlan</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">):</span> <span class="o">...</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">TypeError</span><span class="p">(</span><span class="s">"VLAN ID is not an integer"</span><span class="p">)</span> <span class="o">...</span> <span class="k">elif</span> <span class="ow">not</span> <span class="p">(</span><span class="n">vlan_id</span> <span class="o">&gt;=</span><span class="mi">1</span> <span class="ow">and</span> <span class="n">vlan_id</span> <span class="o">&lt;=</span> <span class="mi">4096</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">"Invalid VLAN ID, which must be between 1-4096"</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="s">"vlan {vlan_id}"</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">vlan_id</span><span class="o">=</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">configure_port_vlan</span><span class="p">(</span><span class="n">interface</span><span class="p">,</span> <span class="n">vlan_id</span><span class="p">):</span> <span class="o">...</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">TypeError</span><span class="p">(</span><span class="s">"VLAN ID is not an integer"</span><span class="p">)</span> <span class="o">...</span> <span class="k">elif</span> <span class="ow">not</span> <span class="p">(</span><span class="n">vlan_id</span> <span class="o">&gt;=</span><span class="mi">1</span> <span class="ow">and</span> <span class="n">vlan_id</span> <span class="o">&lt;=</span> <span class="mi">4096</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">"Invalid VLAN ID, which must be between 1-4096"</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="s">"interface {interface}</span><span class="se">\n</span><span class="s"> switchport access vlan {vlan_id}"</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">interface</span><span class="o">=</span><span class="n">interface</span><span class="p">,</span> <span class="n">vlan_id</span><span class="o">=</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>A common method to to modularize code is building functions. A function can be thought of as a piece of code that can be called to run that single action, and functions can call other functions. Here you can see how the validation functionality is broken out.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">validate_vlan</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">):</span> <span class="o">...</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">TypeError</span><span class="p">(</span><span class="s">"VLAN ID is not an integer"</span><span class="p">)</span> <span class="o">...</span> <span class="k">elif</span> <span class="ow">not</span> <span class="p">(</span><span class="n">vlan_id</span> <span class="o">&gt;=</span><span class="mi">1</span> <span class="ow">and</span> <span class="n">vlan_id</span> <span class="o">&lt;=</span> <span class="mi">4096</span><span class="p">):</span> <span class="o">...</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">"Invalid VLAN ID, which must be between 1-4096"</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>The duplicated functionality is removed from the initial functions, and you can see it was tested to work the same.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">add_vlan</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">):</span> <span class="o">...</span> <span class="n">validate_vlan</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="s">"vlan {vlan_id}"</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">vlan_id</span><span class="o">=</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">configure_port_vlan</span><span class="p">(</span><span class="n">interface</span><span class="p">,</span> <span class="n">vlan_id</span><span class="p">):</span> <span class="o">...</span> <span class="n">validate_vlan</span><span class="p">(</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="k">return</span> <span class="s">"interface {interface}</span><span class="se">\n</span><span class="s"> switchport access vlan {vlan_id}"</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">interface</span><span class="o">=</span><span class="n">interface</span><span class="p">,</span> <span class="n">vlan_id</span><span class="o">=</span><span class="n">vlan_id</span><span class="p">)</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">add_vlan</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span> <span class="n">vlan</span> <span class="mi">500</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">add_vlan</span><span class="p">(</span><span class="mi">5000</span><span class="p">))</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">2</span><span class="p">,</span> <span class="ow">in</span> <span class="n">add_vlan</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">5</span><span class="p">,</span> <span class="ow">in</span> <span class="n">validate_vlan</span> <span class="nb">ValueError</span><span class="p">:</span> <span class="n">Invalid</span> <span class="n">VLAN</span> <span class="n">ID</span><span class="p">,</span> <span class="n">which</span> <span class="n">must</span> <span class="n">be</span> <span class="n">between</span> <span class="mi">1</span><span class="o">-</span><span class="mi">4096</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">configure_port_vlan</span><span class="p">(</span><span class="s">"GigabitEthernet1/0/1"</span><span class="p">,</span> <span class="mi">50</span><span class="p">))</span> <span class="n">interface</span> <span class="n">GigabitEthernet1</span><span class="o">/</span><span class="mi">0</span><span class="o">/</span><span class="mi">1</span> <span class="n">switchport</span> <span class="n">access</span> <span class="n">vlan</span> <span class="mi">50</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">add_vlan</span><span class="p">(</span><span class="s">"500"</span><span class="p">))</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">2</span><span class="p">,</span> <span class="ow">in</span> <span class="n">add_vlan</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">3</span><span class="p">,</span> <span class="ow">in</span> <span class="n">validate_vlan</span> <span class="nb">TypeError</span><span class="p">:</span> <span class="n">VLAN</span> <span class="n">ID</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">an</span> <span class="n">integer</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>Each time you need to validate a VLAN ID, you simply can call the same function. If at a later date you decided that you wanted to validate the VLAN name for length as well, you can simply add that testing to the one function, instead of having to remember all of the multiple places the code can be.</p> <h2 id="dry-data">DRY Data</h2> <p>You will note in the official definition, there is reference to “knowledge”, which as quoted from <a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">Pragmatic Programmer</a></p> <blockquote> <p>“A system’s knowledge is far broader than just its code. It refers to database schemas, test plans, the build system, even documentation”.</p> </blockquote> <p>Simply put, code is not the only thing DRY applies to. One example understood by network engineers is managing data for a subnet mask. I often run into solutions of data and associated Jinja files that similar to:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">subnets</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">network</span><span class="pi">:</span> <span class="s">10.1.1.0</span> <span class="na">cidr</span><span class="pi">:</span> <span class="m">24</span> <span class="na">subnet_mask</span><span class="pi">:</span> <span class="s">255.255.255.0</span> <span class="na">wildcard_mask</span><span class="pi">:</span> <span class="s">0.0.0.255</span> <span class="pi">-</span> <span class="na">network</span><span class="pi">:</span> <span class="s">10.1.2.0</span> <span class="na">cidr</span><span class="pi">:</span> <span class="m">24</span> <span class="na">subnet_mask</span><span class="pi">:</span> <span class="s">255.255.255.0</span> <span class="na">wildcard_mask</span><span class="pi">:</span> <span class="s">0.0.0.255</span> </code></pre></div></div> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">if</span> <span class="nv">ansible_network_os</span> <span class="o">=</span> <span class="s2">"ios"</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'add'</span><span class="p">,</span> <span class="nv">1</span> <span class="p">)</span> <span class="cp">}}</span> <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'subnet_mask'</span><span class="p">]</span> <span class="cp">}}</span><span class="c">{# example result: 10.1.1.1 255.255.255.0 #}</span> <span class="cp">{%</span> <span class="nv">elif</span> <span class="nv">ansible_network_os</span> <span class="o">=</span> <span class="s2">"nxos"</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'add'</span><span class="p">,</span> <span class="nv">1</span> <span class="p">)</span> <span class="cp">}}</span>/<span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'cidr'</span><span class="p">]</span> <span class="cp">}}</span><span class="c">{# example result: 10.1.1.1/24 #}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> </code></pre></div></div> <p>This solution certainly has benefits, however, given the sheer amount of subnets being managed, it is likely at some point there will be a user that creates a <code class="highlighter-rouge">cidr</code> of <code class="highlighter-rouge">25</code> and <code class="highlighter-rouge">subnet_mask</code> as <code class="highlighter-rouge">255.255.255.0</code>. However, the subnet and wildcard mask can be ascertained from the <code class="highlighter-rouge">cidr</code>, and maintaining the data multiple times is superfluous and error prone. The one-time cost of building a translation ensures that you are not repeating yourself and not falling into the issues being described here.</p> <p>Instead you can manage your data a bit more DRY, such as:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">subnets</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">network</span><span class="pi">:</span> <span class="s">10.1.1.0/24</span> <span class="pi">-</span> <span class="na">network</span><span class="pi">:</span> <span class="s">10.1.2.0/24</span> </code></pre></div></div> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">if</span> <span class="nv">ansible_network_os</span> <span class="o">=</span> <span class="s2">"ios"</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'add'</span><span class="p">,</span> <span class="nv">1</span> <span class="p">)</span> <span class="cp">}}</span> <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'netmask'</span><span class="p">)</span> <span class="cp">}}</span><span class="c">{# example result: 10.1.1.1 255.255.255.0 #}</span> <span class="cp">{%</span> <span class="nv">elif</span> <span class="nv">ansible_network_os</span> <span class="o">=</span> <span class="s2">"nxos"</span> <span class="cp">%}</span> ip address <span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'add'</span><span class="p">,</span> <span class="nv">1</span> <span class="p">)</span> <span class="cp">}}</span>/<span class="cp">{{</span> <span class="nv">subnet</span><span class="p">[</span><span class="s1">'network'</span><span class="p">]</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'prefix'</span><span class="p">)</span> <span class="cp">}}</span><span class="c">{# example result: 10.1.1.1/24 #}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> </code></pre></div></div> <p>While this is a trivial example, the potential issue with data duplication is compounded for every feature that needs to be configured on a given device.</p> <h2 id="getting-wet-in-a-dry-world">Getting WET in a DRY World</h2> <p>If DRY is not repeating any piece of knowledge, the reverse would be “Write Every Time” (WET). Now why would you want to repeat yourself? Well, as always, it depends, and there are always design considerations that should be taken into account.</p> <p>As an example, a common pattern we at NTC end up seeing is customers that build a single Jinja template for IOS, NXOS, and EOS. Since these three OS’s have a lot of commonalities, from a DRY perspective you may be inclined to write your templates in consolidated templates.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span> ├── bgp.j2 ├── ntp.j2 └── vlan.j2 0 directories, 3 files </code></pre></div></div> <p>With an example Jinja file such as</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">for</span> <span class="nv">vlan</span> <span class="ow">in</span> <span class="nv">vlans</span> <span class="cp">%}</span> vlan <span class="cp">{{</span> <span class="nv">vlan</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span> <span class="cp">}}</span> name <span class="cp">{{</span> <span class="nv">vlan</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> ! </code></pre></div></div> <p>This works out well for this case, however, as you move on to more complicated use cases, such as BGP, you will end with greater differences between the various OS’s.</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">{%</span> <span class="k">if</span> <span class="nv">ansible_network_os</span> <span class="o">==</span> <span class="s1">'ios'</span> <span class="cp">%}</span> router bgp <span class="cp">{{</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'asn'</span><span class="p">]</span> <span class="cp">}}</span> bgp router-id <span class="cp">{{</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span> <span class="cp">}}</span> bgp log-neighbor-changes <span class="cp">{%</span> <span class="k">for</span> <span class="nv">neighbor</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'neighbors'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">}}</span> remote-as <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'asn'</span><span class="p">]</span> <span class="cp">}}</span> neighbor <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">}}</span> description <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'description'</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> address-family ipv4 <span class="cp">{%</span> <span class="k">for</span> <span class="nv">net</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'networks'</span><span class="p">]</span> <span class="cp">%}</span> network <span class="cp">{{</span> <span class="nv">net</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'network'</span><span class="p">)</span> <span class="cp">}}</span> mask <span class="cp">{{</span> <span class="nv">net</span> <span class="o">| </span><span class="nf">ipaddr</span><span class="p">(</span><span class="s1">'netmask'</span><span class="p">)</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">neighbor</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'neighbors'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">}}</span> activate <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> exit-address-family <span class="c">{# -------------- SWITCH TO NXOS -------------- #}</span> <span class="cp">{%</span> <span class="nv">elif</span> <span class="nv">ansible_network_os</span> <span class="o">==</span> <span class="s1">'nxos'</span> <span class="cp">%}</span> router bgp <span class="cp">{{</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'asn'</span><span class="p">]</span> <span class="cp">}}</span> router-id <span class="cp">{{</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span> <span class="cp">}}</span> address-family ipv4 unicast <span class="cp">{%</span> <span class="k">for</span> <span class="nv">net</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'networks'</span><span class="p">]</span> <span class="cp">%}</span> network <span class="cp">{{</span> <span class="nv">net</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">for</span> <span class="nv">neighbor</span> <span class="ow">in</span> <span class="nv">bgp</span><span class="p">[</span><span class="s1">'neighbors'</span><span class="p">]</span> <span class="cp">%}</span> neighbor <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'ip'</span><span class="p">]</span> <span class="cp">}}</span> remote-as <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'asn'</span><span class="p">]</span> <span class="cp">}}</span> description <span class="cp">{{</span> <span class="nv">neighbor</span><span class="p">[</span><span class="s1">'description'</span><span class="p">]</span> <span class="cp">}}</span> address-family ipv4 unicast <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span> </code></pre></div></div> <p>As you can see, it very quickly get’s complicated. A few basic options, each with pros and cons.</p> <ul> <li>Keep a flat structure, as originally shown, and leverage the similarities of the multiple OS’s to alleviate re-writing them, but add complexity to the use cases where there are differences.</li> <li>Keep a nested structure, one for each OS, and have the code copied over from one OS to the other, where it is the same configuration.</li> <li>Attempt to merge the two, and only leverage OS folder when complicated, and a default folder when not.</li> </ul> <p>In this use case, my personal preference is to have a well defined OS structure as the next developer can review the tree structure (as depicted in the second option), and likely figure out where the configuration templates should go. Essentially, make it easier for the next developer to understand the layout, rather than worry about the duplication of code.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span> ├── eos │   ├── bgp.j2 │   ├── ntp.j2 │   └── vlan.j2 ├── ios │   ├── bgp.j2 │   ├── ntp.j2 │   └── vlan.j2 └── nxos ├── bgp.j2 ├── ntp.j2 └── vlan.j2 3 directories, 9 files </code></pre></div></div> <p>Situations like this will come up all the time, and you will simply have to use experience/intuition to understand the impact and make the best decision from there.</p> <h2 id="conclusion">Conclusion</h2> <p>Managing your code or data should in most use cases be done in a DRY manner, as this method cuts down on errors and drives computational optimizations. That being said, there are some design considerations where it may make sense to take a WET approach.</p> <p>-Ken</p>Ken CelenzaThis is part of a series of posts focused on Network Automation Principles.Metrics for Everyone2019-12-31T00:00:00+00:002019-12-31T00:00:00+00:00https://blog.networktocode.com/post/Metrics<p>This year we heard from a lot of folks that have started down their network automation journey. They are seeing initial success, but are finding it difficult to get buy-in from other teams, management, or both. Although frustrating, this experience is not unique to network automation.</p> <p>DevOps teams have largely experienced the same issue, and looking back to the beginning of DevOps there was a framework put together by John Willis and Damon Edwards in 2010 known as <a href="https://blog.chef.io/what-devops-means-to-me/">CAMS</a>. This framework set out to define the key pillars of DevOps:</p> <ul> <li>Culture</li> <li>Automation</li> <li>Measurement</li> <li>Sharing</li> </ul> <p>Using this framework we can break down the buy-in problem (and finally get to the point of the blog, which is metrics):</p> <p>You or your team have built some piece of automation, and are rightfully excited about it. The culture of your organization is however resistant to change. There are a variety of reasons for this resistance: long history of doing things manually, the expense of building and learning new things, poor results from similar efforts in the past.</p> <h2 id="know-your-audience">Know your audience</h2> <p>Measuring the impact of your automation provides tangible data to combat resistance. It is important however to have the right data for your audience. There are layers to any organization, and each layer has drivers that are relevant to their goals.</p> <p>For the purpose of this blog we will define three layers of the organization and show examples of metrics that may allay the concerns of that audience - know your audience.</p> <h3 id="individual-contributors-engineers">Individual Contributors (Engineers)</h3> <p>This may be network engineers, operations, architects, etc. These are the people most directly related to doing the work. Their primary concerns tend to be:</p> <ul> <li>Level of effort to learn something new</li> <li>Time available to do something different</li> <li>Reliability and performance of solutions</li> </ul> <p>The metrics we want to show here revolve around the state of the infrastructure and the health of the overall solution. What systems are being affected? What failures are we seeing? How is the performance of the automation?</p> <p><img src="../../../static/images/blog_posts/metrics/metrics_1.png" alt="Metrics 1" /> <img src="../../../static/images/blog_posts/metrics/metrics_2.png" alt="Metrics 2" /></p> <h3 id="direct-management">Direct Management</h3> <p>This level of the organization sits in the middle, balancing the drivers and concerns of the others</p> <ul> <li>Capacity and expenditure of the teams resources</li> <li>Well being of the team</li> <li>Ensuring business objectives are articulated bi-bidirectionally and achieved</li> </ul> <p>At this middle level we want to show the impact of automation on the team, the adoption of automation by other teams, and the impact on the business.</p> <p><img src="../../../static/images/blog_posts/metrics/metrics_3.png" alt="Metrics 3" /> <img src="../../../static/images/blog_posts/metrics/metrics_4.png" alt="Metrics 4" /></p> <h3 id="senior-leadership">Senior Leadership</h3> <p>The focus at the top of the organization is big picture - the overarching goals of the company</p> <ul> <li>Overall strategy</li> <li>Financial implications</li> <li>Long term growth and stability</li> </ul> <p><img src="../../../static/images/blog_posts/metrics/metrics_5.png" alt="Metrics 5" style="margin: 0 auto; display: block;" /></p> <h2 id="conclusion">Conclusion</h2> <p>It is important to empathize with all aspects of the organization and provide them the data they need to make informed decisions. DevOps is a cultural shift, as organizations that have started down this path are at various stages of maturity. Data drives business decisions, and the more relevant the data, the better the decisions.</p> <p>The 2018 “<a href="https://www.thinkahead.com/wp-content/uploads/2018/10/State-of-DevOps-Report.pdf">State of DevOps Report</a>” speaks to that journey, and has some great insight:</p> <blockquote> <p>When it comes to measurement, we have found that expansion from teams to departments manifests as a shift from manually gathered IT system metrics to automated measurement of business objectives. The most sophisticated teams we’ve seen not only improved their IT processes and practices, but also managed to focus on delivering business value rather than just technology. These teams have applied their existing cultures of automation and measurement to business objectives.</p> </blockquote> <p>-Rick (@shermdog01)</p>Rick ShermanThis year we heard from a lot of folks that have started down their network automation journey. They are seeing initial success, but are finding it difficult to get buy-in from other teams, management, or both. Although frustrating, this experience is not unique to network automation.Exploring Jinja2 Variable Syntax in Ansible2019-12-24T00:00:00+00:002019-12-24T00:00:00+00:00https://blog.networktocode.com/post/Exploring-Jinja-Variable-Syntax-in-Ansible<p>When working with Ansible <code class="highlighter-rouge">playbooks</code> and <code class="highlighter-rouge">jinja2</code>, you might notice different jinja2 syntax to access data. Some use square brackets (<code class="highlighter-rouge">backup_commands['ios']</code>) while others use dot notation (<code class="highlighter-rouge">backup_commands.ios</code>) when accessing data.</p> <p>Both ways are valid and work fairly well, <strong>but</strong> some would say it’s safer to use <code class="highlighter-rouge">[]</code> bracket rather than <code class="highlighter-rouge">.</code> dot notation to access data in a production environment. For demo purposes, I tend to use the<code class="highlighter-rouge">.</code> notation just to type quicker but that won’t always work if accessing data from a variable.</p> <blockquote> <p>Note: Other reasons to use square brackets:</p> <ul> <li>When a variable name includes a hyphen (<code class="highlighter-rouge">-</code>)</li> <li>When dynamically generating a dictionary key</li> </ul> </blockquote> <p>Let’s look at a few examples and see why it’s not always possible to access data using the dot(<code class="highlighter-rouge">.</code>) notation and it’s always possible to access data using the square brackets(<code class="highlighter-rouge">[]</code>).</p> <h2 id="using-ansible-to-access-dictionary-values">Using Ansible to Access Dictionary Values</h2> <p>First let’s test this playbook called <code class="highlighter-rouge">brackets_vs_dot.yml</code> and run it.</p> <p>A variable has been created under <code class="highlighter-rouge">vars</code> called <code class="highlighter-rouge">backup_commands</code> and nested, there are a few <code class="highlighter-rouge">{ "key": "value" }</code> pairs with different vendors as keys and commands as values to access.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">backup_commands</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">junos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">configuration"</span> </code></pre></div></div> <p>Four <code class="highlighter-rouge">debug</code> tasks have also been added that will print out each <code class="highlighter-rouge">key</code> in a nested <code class="highlighter-rouge">dict</code> of <code class="highlighter-rouge">dicts</code>. Two of the tasks will use bracket <code class="highlighter-rouge">[]</code> notation and the other two will use dot <code class="highlighter-rouge">.</code> notation.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Accessing values through dict keys</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">local</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">backup_commands</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">junos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">configuration"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands['ios']</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands['nxos']</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands.ios</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands.nxos</span> </code></pre></div></div> <p>After running the playbook, you should see the following output with the values being accessed using the <code class="highlighter-rouge">debug</code> module.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ntc@jump-host:ansible<span class="nv">$ </span>brackets_vs_dot.yml PLAY <span class="o">[</span>Accessing values through dict keys] <span class="k">**********************************************************************************************************************************************************************************</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands['ios']"</span>: <span class="s2">"show run"</span> <span class="o">}</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands['nxos']"</span>: <span class="s2">"show run"</span> <span class="o">}</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands.ios"</span>: <span class="s2">"show run"</span> <span class="o">}</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands.nxos"</span>: <span class="s2">"show run"</span> <span class="o">}</span> PLAY RECAP <span class="k">*****************************************************************************************************************************************************************************************************************</span> localhost : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>Based on the output you can see that both bracket <code class="highlighter-rouge">[]</code> and <code class="highlighter-rouge">.</code> dot notation worked just fine with the same output results.</p> <p>However, let’s say this time we want to access that same data, but through a variable. The results on this test will be a little different and will show why the <code class="highlighter-rouge">.</code> notation doesn’t always work.</p> <p>This time <code class="highlighter-rouge">vars_prompt</code> has been added in a second play with a variable <code class="highlighter-rouge">vendor_os</code> to allow the user to pick an <code class="highlighter-rouge">os</code> and access the data through the <code class="highlighter-rouge">debug</code> tasks.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">vars_prompt</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">vendor_os</span> <span class="na">prompt</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Enter</span><span class="nv"> </span><span class="s">Vendor</span><span class="nv"> </span><span class="s">OS"</span> <span class="na">private</span><span class="pi">:</span> <span class="s">no</span> </code></pre></div></div> <p>In this second play two tasks have been added using <code class="highlighter-rouge">[]</code> brackets and <code class="highlighter-rouge">.</code> dot notation to access the data based on the variable value entered through the prompt.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Accessing values through variables</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">local</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">vars_prompt</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">vendor_os</span> <span class="na">prompt</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Enter</span><span class="nv"> </span><span class="s">Vendor</span><span class="nv"> </span><span class="s">OS"</span> <span class="na">private</span><span class="pi">:</span> <span class="s">no</span> <span class="na">vars</span><span class="pi">:</span> <span class="na">backup_commands</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">junos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">configuration"</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands[vendor_os]</span> <span class="pi">-</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">var</span><span class="pi">:</span> <span class="s">backup_commands.vendor_os</span> </code></pre></div></div> <h3 id="accessing-values-through-variables">Accessing values through variables</h3> <p>Pay close attention to the second <code class="highlighter-rouge">debug</code> tasks with the results of <code class="highlighter-rouge">"VARIABLE IS NOT DEFINED!"</code>.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Enter Vendor OS: junos </code></pre></div></div> <p>After entering <code class="highlighter-rouge">junos</code> in the prompt the following output show be displayed:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PLAY <span class="o">[</span>Accessing values through variables] <span class="k">**********************************************************************************************************************************************************************************</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands[vendor_os]"</span>: <span class="s2">"show configuration"</span> <span class="o">}</span> TASK <span class="o">[</span>debug] <span class="k">***************************************************************************************************************************************************************************************************************</span> ok: <span class="o">[</span>localhost] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"backup_commands.vendor_os"</span>: <span class="s2">"VARIABLE IS NOT DEFINED!"</span> <span class="o">}</span> PLAY RECAP <span class="k">*****************************************************************************************************************************************************************************************************************</span> localhost : <span class="nv">ok</span><span class="o">=</span>6 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>If you’re still not convinced why we should stick to using <code class="highlighter-rouge">[]</code> bracket notation vs <code class="highlighter-rouge">.</code> dot notation then let’s try this out with <code class="highlighter-rouge">jinja2</code>.</p> <h2 id="comparing-dot-notation-vs-square-brackets-using-jinja2">Comparing Dot Notation vs Square Brackets using Jinja2</h2> <p>For this test the main tool being used is <a href="https://td4a.now.sh">Template Designer for Automation</a>, feel free to use this data and template to test it out yourself:</p> <h3 id="data">DATA</h3> <p>Using the same data structure from the previous Ansible test.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">backup_commands</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">run"</span> <span class="na">junos</span><span class="pi">:</span> <span class="s2">"</span><span class="s">show</span><span class="nv"> </span><span class="s">configuration"</span> </code></pre></div></div> <h3 id="jinja2-template---accessing-values-through-dict-keys">JINJA2 TEMPLATE - Accessing values through dict keys</h3> <p>A similar test to the first play of Ansible will be ran. This time instead of <code class="highlighter-rouge">debug</code> tasks, build a <code class="highlighter-rouge">jinja2</code> template and add it in the <code class="highlighter-rouge">TEMPLATE</code> column of <a href="https://td4a.now.sh">td4a</a>.</p> <p>In this example, each <code class="highlighter-rouge">key</code> will be accessed with square(<code class="highlighter-rouge">[]</code>) bracket and dot (<code class="highlighter-rouge">.</code>) notation.</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Accessing Keys Bracket Notation: ios: <span class="cp">{{</span> <span class="nv">backup_commands</span><span class="p">[</span><span class="s1">'ios'</span><span class="p">]</span> <span class="cp">}}</span> nxos: <span class="cp">{{</span> <span class="nv">backup_commands</span><span class="p">[</span><span class="s1">'nxos'</span><span class="p">]</span> <span class="cp">}}</span> junos: <span class="cp">{{</span> <span class="nv">backup_commands</span><span class="p">[</span><span class="s1">'junos'</span><span class="p">]</span> <span class="cp">}}</span> Dot Notation: ios: <span class="cp">{{</span> <span class="nv">backup_commands.ios</span> <span class="cp">}}</span> nxos: <span class="cp">{{</span> <span class="nv">backup_commands.nxos</span> <span class="cp">}}</span> junos: <span class="cp">{{</span> <span class="nv">backup_commands.junos</span> <span class="cp">}}</span> </code></pre></div></div> <p>After adding the following <code class="highlighter-rouge">jinja2</code> data into the <code class="highlighter-rouge">TEMPLATE</code> column, press the <code class="highlighter-rouge">RENDER</code> button on the top right of the middle column to view the results.</p> <p><img src="../../../static/images/blog_posts/render.png" alt="" /></p> <h3 id="result">RESULT</h3> <p>You should see the following results, as expected from the <code class="highlighter-rouge">jinja2</code> template showing that it has successfully accessed each value using both <code class="highlighter-rouge">[]</code> brackets and <code class="highlighter-rouge">.</code> notation.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">Accessing Keys</span> <span class="na">Bracket Notation</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">junos</span><span class="pi">:</span> <span class="s">show configuration</span> <span class="na">Dot Notation</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">junos</span><span class="pi">:</span> <span class="s">show configuration</span> </code></pre></div></div> <h3 id="jinja2-template---accessing-values-through-variables">JINJA2 TEMPLATE - Accessing values through variables</h3> <p>This time run a test with <code class="highlighter-rouge">jinja2</code> for loops, the first one accessing the values through <code class="highlighter-rouge">[]</code> brackets and the second one commented out with <code class="highlighter-rouge">.</code> notation for now so it doesn’t dispaly an error.</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Accessing Variables Bracket Notation: <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vendor_os</span> <span class="ow">in</span> <span class="nv">backup_commands.keys</span><span class="p">()</span> <span class="cp">%}</span> <span class="cp">{{</span> <span class="nv">vendor_os</span> <span class="cp">}}</span>: <span class="cp">{{</span> <span class="nv">backup_commands</span><span class="p">[</span><span class="nv">vendor_os</span><span class="p">]</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> Dot Notation: <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vendor_os</span> <span class="ow">in</span> <span class="nv">backup_commands.keys</span><span class="p">()</span> <span class="cp">%}</span> <span class="c">{#{{ vendor_os }}: {{ backup_commands.vendor_os }}#}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> </code></pre></div></div> <p>After pressing on the <code class="highlighter-rouge">RENDER</code> button again –&gt; <img src="../../../static/images/blog_posts/render.png" alt="" /> the following output should show up:</p> <h3 id="result-1">RESULT</h3> <p>The first two tests don’t change, the second two show the output as expected using the <code class="highlighter-rouge">[]</code> bracket and the <code class="highlighter-rouge">.</code> notation doesn’t show anything since it’s commented out.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">Accessing Keys</span> <span class="na">Bracket Notation</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">junos</span><span class="pi">:</span> <span class="s">show configuration</span> <span class="na">Dot Notation</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">junos</span><span class="pi">:</span> <span class="s">show configuration</span> <span class="s">Accessing Variables</span> <span class="na">Bracket Notation</span><span class="pi">:</span> <span class="na">ios</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">nxos</span><span class="pi">:</span> <span class="s">show run</span> <span class="na">junos</span><span class="pi">:</span> <span class="s">show configuration</span> <span class="na">Dot Notation</span><span class="pi">:</span> </code></pre></div></div> <p>Finally it’s time to test the dot notation so remove the <code class="highlighter-rouge">{##}</code> comments from the <code class="highlighter-rouge">jinja2</code> template to view the results using the <code class="highlighter-rouge">.</code> notation.</p> <div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Dot Notation: <span class="cp">{%</span> <span class="k">for</span> <span class="nv">vendor_os</span> <span class="ow">in</span> <span class="nv">backup_commands.keys</span><span class="p">()</span> <span class="cp">%}</span> <span class="cp">{{</span> <span class="nv">vendor_os</span> <span class="cp">}}</span>: <span class="cp">{{</span> <span class="nv">backup_commands.vendor_os</span> <span class="cp">}}</span> <span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span> </code></pre></div></div> <p>After pressing on the <code class="highlighter-rouge">RENDER</code> button one last time and viewing the results:</p> <p>You should see the following error: <code class="highlighter-rouge">Message: Issue found loading template. Details: Object has no attribute 'vendor_os' Line number: 22</code></p> <p><img src="../../../static/images/blog_posts/error.png" alt="" /></p> <p>Finally it has been proven that for good practice and sake of muscle memory, that maybe we should stick to using <code class="highlighter-rouge">[]</code> square brackets rather than <code class="highlighter-rouge">.</code> dot notation because you will always be able to access the data whether if it’s a key or a variable without any errors.</p> <p>-Hector</p>Hector IsazaWhen working with Ansible playbooks and jinja2, you might notice different jinja2 syntax to access data. Some use square brackets (backup_commands['ios']) while others use dot notation (backup_commands.ios) when accessing data.