Jekyll2020-03-31T18:10:27+00:00https://blog.networktocode.com/feed.xmlThe NTC MagNetwork to Codeinfo@networktocode.comHow to Monitor Your VPN Infrastructure with Netmiko, NTC-Templates, and a Time Series Database2020-03-30T00:00:00+00:002020-03-30T00:00:00+00:00https://blog.networktocode.com/post/using_python_and_telegraf_for_metrics<p>With many people being asked to work from home, we have heard several customers looking to enhance the visibiility and monitoring of their VPN infrastructure. In this post I will show you how you can quickly collect information from your Cisco ASA firewall leveraging <a href="https://github.com/ktbyers/netmiko">Netmiko</a>, <a href="https://github.com/networktocode/ntc-templates">NTC-Templates</a> (TextFSM), combined with Telegraf/Prometheus/Grafana. The approach would work on other network devices, not just Cisco ASAs. Considering the recent demand for ASA information, we will use this as an example.</p> <p>Here is what the data flow will look like:</p> <p><img src="../../../static/images/blog_posts/prometheus_asa_visibility/data_flow.png" alt="App Flow" /></p> <ul> <li>Users will connect to the ASA for remote access VPN services</li> <li>Python: Collects information from the device via CLI and gets structured data by using a <strong>NEW</strong> template to parse the CLI output. This is presented via stdout in Influx data format</li> <li><a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a>: Generic collector that has multiple plugins to ingest data and that can send data to many databases out there. <ul> <li>INPUT: Execute the Python script every 60s and read the results from stdout.</li> <li>OUTPUT: Expose the data over HTTP in a format compatible with Prometheus</li> </ul> </li> <li><a href="https://prometheus.io/">Prometheus</a>: Time Series DataBase (TSDB). Collects the data from Telegraf over HTTP, stores it, and exposes an API to query the data</li> <li><a href="https://grafana.com/">Grafana</a>: Solution to build dashboards, natively support Prometheus to query data.</li> </ul> <blockquote> <p>An alternative to creating this Python script, you could have looked at using the Telegraf SNMP plugin as well. An SNMP query would be quicker than using SSH and getting data if you want basic counts. In this you will see that you can get custom metrics into a monitoring solution without having to use only SNMP.</p> </blockquote> <h2 id="execution-of-python">Execution of Python</h2> <p>If executing just the Python script without being executed by Telegraf, this is what you would see:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python3 asa_anyconnect_to_telegraf.py <span class="nt">--host</span> 10.250.0.63 asa <span class="nv">connected_users</span><span class="o">=</span>1i,anyconnect_licenses<span class="o">=</span>2i </code></pre></div></div> <p>This data will then get transformed by Telegraf into an output that is usable by Prometheus. It is possible to remove the requirement for Telegraf and have Python create the Prometheus Metrics. We wanted to keep the Python execution as simple as possible. To use the <a href="https://github.com/prometheus/client_python">prometheus_client</a> library check out their Github page.</p> <h2 id="python-script">Python Script</h2> <p>In this post we have the following components being used:</p> <ul> <li>Python: <ul> <li><a href="https://pynet.twb-tech.com/blog/automation/netmiko.html">Netmiko</a> to SSH into an ASA, gather command output, and leverage the corresponding NTC Template</li> <li><a href="https://github.com/networktocode/ntc-templates">NTC Template</a> which is a TextFSM template for parsing raw text output into structured data</li> </ul> </li> <li>Telegraf: Takes the output of the Python script as an input and translates it to Prometheus metrics as an output</li> </ul> <h3 id="python-requirements">Python Requirements</h3> <p>The Python script below will have the following requirements set up before hand:</p> <ul> <li>ENV Variables for authentication into the ASA <ul> <li><strong>ASA_USER</strong>: Username to log into the ASA</li> <li><strong>ASA_PASSWORD</strong>: Password to log into the ASA</li> <li><strong>ASA_SECRET</strong> (Optional): Enable password for the ASA, if left undefined will pick up the <strong>ASA_PASSWORD</strong> variable</li> </ul> </li> <li>Required Python Packages: <ul> <li><em>Netmiko</em>: For SSH and parsing</li> <li><em>Click</em>: For argument handling - to get the hostname/IP address of the ASA</li> </ul> </li> <li>Github Repository for NTC-Templates setup in one of two ways: <ul> <li>Cloned to user home directory <code class="highlighter-rouge">cd ~</code> and <code class="highlighter-rouge">git clone https://github.com/networktocode/ntc-templates.git</code></li> <li><strong>NET_TEXTFSM</strong> Env variable set <code class="highlighter-rouge">NET_TEXTFSM=/path/to/ntc-templates/templates/</code></li> </ul> </li> </ul> <blockquote> <p>The specific template is the newer template for <a href="https://github.com/networktocode/ntc-templates/blob/master/templates/cisco_asa_show_vpn-sessiondb_anyconnect.textfsm">cisco asa show vpn-sessiondb anyconnect</a> introduced March 18, 2020</p> </blockquote> <h3 id="python-code">Python Code</h3> <p>There are two functions used in this quick script:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">""" (c) 2020 Network to Code Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. You may obtain a copy of the License at http: // www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Python application to gather metrics from Cisco ASA firewall and export them as a metric for Telegraf """</span> <span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">count</span> <span class="kn">import</span> <span class="nn">os</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="nn">re</span> <span class="kn">import</span> <span class="nn">click</span> <span class="kn">from</span> <span class="nn">netmiko</span> <span class="kn">import</span> <span class="n">ConnectHandler</span> <span class="k">def</span> <span class="nf">print_influx_metrics</span><span class="p">(</span><span class="n">data</span><span class="p">):</span> <span class="s">""" The print_influx_metrics function takes the data collected in a dictionary format and prints out each of the necesary components on a single line, which matches the Influx data format. Args: data (dictionary): Dictionary of the results to print out for influx """</span> <span class="n">data_string</span> <span class="o">=</span> <span class="s">""</span> <span class="n">cnt</span> <span class="o">=</span> <span class="n">count</span><span class="p">()</span> <span class="k">for</span> <span class="n">measure</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="k">if</span> <span class="nb">next</span><span class="p">(</span><span class="n">cnt</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span> <span class="n">data_string</span> <span class="o">+=</span> <span class="s">","</span> <span class="n">data_string</span> <span class="o">+=</span> <span class="n">f</span><span class="s">"{measure}={value}i"</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s">"asa {data_string}"</span><span class="p">)</span> <span class="k">return</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">get_anyconnect_license_count</span><span class="p">(</span><span class="n">version_output</span><span class="p">):</span> <span class="s">""" Searches through the `show version` output to find all instances of the license and gets the output into integers to get a license count. Since there could be multiple ASAs in a cluster or HA pair, it is necessary to gather multiple data points for the license count that the ASAs are licensed for. This function uses regex to find all of the instances and returns the total count based on the `show version` command output. Args: version_output (String): Output from Cisco ASA `show version` """</span> <span class="n">pattern</span> <span class="o">=</span> <span class="s">r"AnyConnect\s+Premium\s+Peers\s+:\s+(\d+)"</span> <span class="n">re_list</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">version_output</span><span class="p">)</span> <span class="n">total_licenses</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">license_count</span> <span class="ow">in</span> <span class="n">re_list</span><span class="p">:</span> <span class="n">total_licenses</span> <span class="o">+=</span> <span class="nb">int</span><span class="p">(</span><span class="n">license_count</span><span class="p">)</span> <span class="k">return</span> <span class="n">total_licenses</span> <span class="c1"># Add parsers for output of data types </span><span class="o">@</span><span class="n">click</span><span class="o">.</span><span class="n">command</span><span class="p">()</span> <span class="o">@</span><span class="n">click</span><span class="o">.</span><span class="n">option</span><span class="p">(</span><span class="s">"--host"</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">"Required - Host to connect to"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">host</span><span class="p">):</span> <span class="s">""" Main code execution """</span> <span class="c1"># Get ASA connection Information </span> <span class="k">try</span><span class="p">:</span> <span class="n">username</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"ASA_USER"</span><span class="p">]</span> <span class="n">password</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"ASA_PASSWORD"</span><span class="p">]</span> <span class="n">secret</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"ASA_SECRET"</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"ASA_PASSWORD"</span><span class="p">])</span> <span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">"Unable to find Username or Password in environment variables"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"Please verify that ASA_USER and ASA_PASSWORD are set"</span><span class="p">)</span> <span class="n">sys</span><span class="o">.</span><span class="nb">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># Setup connection information and connect to host </span> <span class="n">cisco_asa_device</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"host"</span><span class="p">:</span> <span class="n">host</span><span class="p">,</span> <span class="s">"username"</span><span class="p">:</span> <span class="n">username</span><span class="p">,</span> <span class="s">"password"</span><span class="p">:</span> <span class="n">password</span><span class="p">,</span> <span class="s">"secret"</span><span class="p">:</span> <span class="n">secret</span><span class="p">,</span> <span class="s">"device_type"</span><span class="p">:</span> <span class="s">"cisco_asa"</span><span class="p">,</span> <span class="p">}</span> <span class="n">net_conn</span> <span class="o">=</span> <span class="n">ConnectHandler</span><span class="p">(</span><span class="o">**</span><span class="n">cisco_asa_device</span><span class="p">)</span> <span class="c1"># Get command output for data collection </span> <span class="n">command</span> <span class="o">=</span> <span class="s">"show vpn-sessiondb anyconnect"</span> <span class="n">command_output</span> <span class="o">=</span> <span class="n">net_conn</span><span class="o">.</span><span class="n">send_command</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">use_textfsm</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c1"># Check for no connected users </span> <span class="k">if</span> <span class="s">"INFO: There are presently no active sessions"</span> <span class="ow">in</span> <span class="n">command_output</span><span class="p">:</span> <span class="n">command_output</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># Get output of "show version" </span> <span class="n">version_output</span> <span class="o">=</span> <span class="n">net_conn</span><span class="o">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">"show version"</span><span class="p">)</span> <span class="c1"># Set data variable for output to Influx format </span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s">"connected_users"</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">command_output</span><span class="p">),</span> <span class="s">"anyconnect_licenses"</span><span class="p">:</span> <span class="n">get_anyconnect_license_count</span><span class="p">(</span><span class="n">version_output</span><span class="p">)}</span> <span class="c1"># Print out the metrics to standard out to be picked up by Telegraf </span> <span class="n">print_influx_metrics</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </code></pre></div></div> <h2 id="telegraf">Telegraf</h2> <p>Now that the data is being output via the stdout of the script, you will need to have an application read this data and transform it. This could be done in other ways as well, but Telegraf has this function built in already.</p> <p>Telegraf will be setup to execute the Python script every minute. Then the output will be transformed by defining the output.</p> <h3 id="telegraf-configuration">Telegraf Configuration</h3> <p>The configuration for this example is as follows:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Globally set tags that shuld be set to meaningful tags for searching inside of a TSDB</span> <span class="o">[</span>agent] <span class="nb">hostname</span> <span class="o">=</span> <span class="s2">"demo"</span> <span class="o">[</span>global_tags] device <span class="o">=</span> <span class="s2">"10.250.0.63"</span> region <span class="o">=</span> <span class="s2">"midwest"</span> <span class="o">[[</span>inputs.exec]] <span class="c">## Interval is how often the execution should occur, here every 1 min (60 seconds)</span> interval <span class="o">=</span> <span class="s2">"60s"</span> <span class="c"># Commands to be executed in list format</span> <span class="c"># To execute against multiple hosts, add multiple entries within the commands</span> commands <span class="o">=</span> <span class="o">[</span> <span class="s2">"python3 asa_anyconnect_to_telegraf.py --host 10.250.0.63"</span> <span class="o">]</span> <span class="c">## Timeout for each command to complete.</span> <span class="c"># Tests in lab environment next to the device with local authentication has been 6 seconds</span> <span class="nb">timeout</span> <span class="o">=</span> <span class="s2">"15s"</span> <span class="c">## Measurement name suffix (for separating different commands)</span> name_suffix <span class="o">=</span> <span class="s2">"_parsed"</span> <span class="c">## Data format to consume.</span> <span class="c">## Each data format has its own unique set of configuration options, read</span> <span class="c">## More about them here:</span> <span class="c">## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md</span> data_format <span class="o">=</span> <span class="s2">"influx"</span> <span class="c"># Output to Prometheus Metrics format</span> <span class="c"># Define the listen port for which TCP port the web server will be listening on. Metrics will be</span> <span class="c"># available at "http://localhost:9222/metrics" in this instance.</span> <span class="c"># There are two versions of metrics and if `metric_version` is omitted then version 1 is used</span> <span class="o">[[</span>outputs.prometheus_client]] listen <span class="o">=</span> <span class="s2">":9222"</span> metric_version <span class="o">=</span> 2 </code></pre></div></div> <h3 id="telegraf-output-example">Telegraf Output Example</h3> <p>Here is what the metrics will look like when exposed, without the default Telegraf information metrics.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># HELP asa_parsed_anyconnect_licenses Telegraf collected metric # TYPE asa_parsed_anyconnect_licenses untyped asa_parsed_anyconnect_licenses{device="10.250.0.63",host="demo",region="midwest"} 2 # HELP asa_parsed_connected_users Telegraf collected metric # TYPE asa_parsed_connected_users untyped asa_parsed_connected_users{device="10.250.0.63",host="demo",region="midwest"} 1 </code></pre></div></div> <p>There are two metrics of <code class="highlighter-rouge">anyconnect_licenses</code> and <code class="highlighter-rouge">connected_users</code> that will get scraped. There are a total of 2 Anyconnect licenses available on this firewall with a single user connected. This can now get scraped by the Prometheus configuration and give insight to your ASA Anyconnect environment.</p> <h2 id="prometheus-installation">Prometheus Installation</h2> <p>There are several of options for installing a Prometheus TSDB (Time Series DataBase)including:</p> <ul> <li>Precompiled binaries for Windows, Mac, and Linux</li> <li>Docker images</li> <li>Building from source</li> </ul> <p>To get more details on installation options take a look at the <a href="https://github.com/prometheus/prometheus">Prometheus Github page</a>.</p> <p>Once installed you can navigate to the Prometheus API query page by going to <code class="highlighter-rouge">http://&lt;prometheus_host&gt;:9090</code>. You will then be presented with a search bar. This is where you can start a query for your metric that you wish to graph, such as start typing <code class="highlighter-rouge">asa</code>. Prometheus will help with an autocomplete set of options in the search bar. Once you have selected what you wish to query you can select <strong>Execute</strong>. This will give you a value at this point. To see what the query looks like over time you can select Graph next to the world console to give you a graph over time. Grafana will then use the same query language to add a graph.</p> <p>Once up and running, you add your Telegraf host to the scraping configuration and Prometheus will start to scrape the HTML page provided and add the associated metrics into its TSDB.</p> <p>A good video tutorial for getting started with Prometheus queries with network equipment can be found on <a href="https://youtu.be/lzppzWGRHGo">YouTube from NANOG 77</a></p> <h2 id="grafana-installation">Grafana Installation</h2> <p>Grafana is the dashboarding component of choice in the open source community. Grafana is able to use several sources to create graphs including modern TSDBs of InfluxDB and Prometheus. With the latest release, Grafana can even use Google Sheets as a datasource.</p> <p>As you get going with Grafana there are <a href="https://grafana.com/grafana/dashboards">pre-built dashboards</a> available for download.</p> <p>TYou will want to download Grafana to get started. There are several installation methods available on their <a href="https://grafana.com/grafana/download">download page</a> including options for:</p> <ul> <li>Linux</li> <li>Windows</li> <li>Mac</li> <li>Docker</li> <li>ARM (Raspberry Pi)</li> </ul> <p>The Prometheus website has an <a href="https://prometheus.io/docs/visualization/grafana/">article that is helpful for getting started</a> with your own Prometheus dashboards.</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/Tr-U6_toDf4" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <p>If the video just above this link does not show up, you can see the video on the Network to Code <a href="https://www.youtube.com/watch?v=Tr-U6_toDf4&amp;feature=youtu.be">YouTube</a>.</p> <p>-Josh</p>Josh VanDeraaWith many people being asked to work from home, we have heard several customers looking to enhance the visibiility and monitoring of their VPN infrastructure. In this post I will show you how you can quickly collect information from your Cisco ASA firewall leveraging Netmiko, NTC-Templates (TextFSM), combined with Telegraf/Prometheus/Grafana. The approach would work on other network devices, not just Cisco ASAs. Considering the recent demand for ASA information, we will use this as an example.Our Top Work from Home Tips2020-03-24T00:00:00+00:002020-03-24T00:00:00+00:00https://blog.networktocode.com/post/work-from-home<p>Whether you’re a seasoned home office pro or new to working from home, there’s no doubt that working in the place you live can present some unique challenges. Here at Network to Code, we’re no strangers to working from home – we’ve had remote workers across the US and EU for several years now. But given the sudden increase in people working from home, our team wanted to share some of our best tips and tricks for being productive outside a traditional office setting.</p> <h3 id="working-from-home-as-an-extrovert-stephen-kiely-network-automation-engineer">Working from home as an extrovert (Stephen Kiely, Network Automation Engineer)</h3> <p>My number one recommendation would be to switch to video calls for all your meetings. Video calls help you engage in conversation more effectively (just make sure to resist the urge to read emails or engage with other distractions during calls).</p> <p>I also recommend finding a way to engage in “water cooler chat” remotely. Since you can’t get up and have conversations with colleagues, make sure you have an appropriate place where these conversations are accepted (Slack channels are great for this). At Network to Code, we also use <a href="https://www.donut.com/pairing/">virtual donuts</a> – a service that sets you up with a new colleague each week for a conversation. Leave work out of these chats and use this as an opportunity to meet your coworkers!</p> <h3 id="communication-is-key-josh-vanderaa-network-automation-engineer">Communication is key (Josh VanDeraa, Network Automation Engineer)</h3> <p>Working from home means setting boundaries, both with the other people you work with and with your colleagues. If possible, try to have a separate workspace to enforce this boundary. When you are working, you are working – communicating this to those you share a space with is key to your success.</p> <p>It’s also critical that you speak up on the communication mediums provided to you (Slack, email, telephone, etc.) Make sure to regularly reach out to others and pay attention to building bonds (something those in office buildings may take for granted).</p> <h3 id="keep-healthy-snacks-around-daniel-himes-network-automation-engineer">Keep healthy snacks around (Daniel Himes, Network Automation Engineer)</h3> <p>Keep something healthy around that you can snack on. You will, at some point during the day, find yourself drawn to the kitchen. Try to make sure it’s much easier to reach for the carrots than donuts.</p> <h3 id="remember-to-take-breaks-jere-julian-managing-director--principle-architect">Remember to take breaks (Jere Julian, Managing Director &amp; Principle Architect)</h3> <p>Don’t forget to take a break every once in a while – you do it in the office and it’s important to make space for free time at home as well. Something as simple as starting laundry can give you the mental break you need, and it probably takes less time than when someone randomly stops by your desk at the office “for a chat.”</p> <p>If you normally go out to lunch when in the office, make sure to completely step away when working from home too. Eat lunch in the kitchen instead of your office. Your mind needs that change of pace and scenery.</p> <h3 id="separate-your-home-and-your-office-and-dont-get-too-comfortable-jeremy-stretch-sr-network-automation-engineer">Separate your home and your office and don’t get too comfortable! (Jeremy Stretch, Sr. Network Automation Engineer)</h3> <p>When you work from home, “work time” and “home time” tend to blend together. It’s crucial to establish a dedicated workspace to help demarcate the two. Dedicating a specific room within your home is ideal, but even if you’re short on space, try carving out a corner of the living room or a specific chair and desk that you reserve for work functions. Be sure to avoid the bedroom: working where you sleep <a href="https://valleysleepcenter.com/5-things-you-shouldnt-do-in-your-bedroom-2/">can cause difficulty sleeping.</a></p> <p>Also be sure to bear in mind that work is still work. Don’t succumb to the temptation to “settle in” at home. If you’ve just recently made the switch from an office job, be sure to keep up your previous morning routine: shower, get dressed, eat breakfast, etc. It also helps to designate a specific start time each day and abide by it. There are no excuses for being late when you work at home!</p> <h3 id="working-from-home-with-kids-and-setting-up-your-home-office-bryan-culver-sr-network-automation-engineer">Working from home with kids and setting up your home office (Bryan Culver, Sr. Network Automation Engineer)</h3> <p>When it comes to working from home with kids, I can’t say this enough times: SCHEDULE! SCHEDULE! SCHEDULE! Everyone else has already keyed into why to split your time for your own mental health but also this isn’t your every-day working from home. Odds are your kids are home now, and for most people both parents still need to work. This is OK. Communicate with your respective bosses and coworkers what your current working situation is and what you’re willing to do to work around it. But you can’t do that without at least identifying what your children’s needs. Younger ones need more structure than older ones, so work on identifying the times you need to be away from your desk. Everyone is going to have to move around schedules and communicating that before it becomes a problem will help diffuse a lot of frustration.</p> <p>Setting up an appropriate workspace is also key. Working on top of milk cartons wouldn’t fly in an office, and the same applies at home. There’s no need to break the bank and there are a bunch of DIY standing desk instructions you can find online. Define your space and respect it. Work with your employer to see if they will help purchase or offset some of these costs as well. If your business has a lot of synchronous conversations (calls, video chats, etc.) invest in better equipment. This means a better microphone, better webcam, and more comfortable headphones. If you have a noisier home office setting, I suggest looking at cardioid style microphones which are very directional (I’m personally a fan of this <a href="https://www.amazon.com/Audio-Technica-ATR2100x-USB-Cardioid-Microphone-ATR/dp/B07ZPBFVKK/ref=dp_ob_title_ce">one</a>—they help cut out audio not directly in front of them.</p> <h2 id="conclusion">Conclusion</h2> <p>We hope these tips have been helpful for those of you who are working from home. Make sure to take care of yourself and those around you! We’ll close with a tip near and dear to all our hearts: If you have a pet (or office manager as we refer to the around these parts) make sure to schedule regular <del>walks</del> meetings. They’ll love it, you’ll love it.</p> <p>-The NTC Team</p>The NTC TeamWhether you’re a seasoned home office pro or new to working from home, there’s no doubt that working in the place you live can present some unique challenges. Here at Network to Code, we’re no strangers to working from home – we’ve had remote workers across the US and EU for several years now. But given the sudden increase in people working from home, our team wanted to share some of our best tips and tricks for being productive outside a traditional office setting.Leveraging Network Automation for Proactive Diagnostics and Visibility2020-03-18T00:00:00+00:002020-03-18T00:00:00+00:00https://blog.networktocode.com/post/leveraging-network-automation-for-proactive-diagnostics-and-visibility<p>By now, COVID-19 has impacted all of us in some way, shape, or form. We truly hope everyone is following the guidelines and recommendations from WHO and their local authorities.</p> <p>Most of our clients have already implemented a mandatory “work from home” policy for the foreseeable future (all of our employees are fully remote too). What we are seeing is a major impact to the technology infrastructure supporting their businesses. Often it is the front-line workers or customers that are consuming the infrastructure.</p> <p>For example, after speaking to just a few of our clients, we’ve heard the following:</p> <ul> <li>There is a greater need for visibility and dashboards for all types of traffic, performance, and capacity for remote and VPN infrastructure. There is increased troubleshooting happening in remote access, security, and VPN infrastructure.</li> <li>There is increased use of VPN concentrators that may not be sized for performance, so they are actively disconnecting users that are idle for more than a certain period of time.</li> <li>There is increased use of collaboration systems like WebEx Teams, Slack, and MS Teams.</li> <li>With the general public at home, video games and streaming are putting even more demand on the network infrasructure and supporting systems.</li> </ul> <p>A common approach to these problems we’ve taken in the past (way before COVID-19) is to automate proactive troubleshooting and diagnostics ensuring there isn’t degraded performance and also to ensure there is enough capacity on remote access, security, wireless, and collaboration systems. Automating these types of tasks can easily be integrated to Chat programs as well ensuring anyone on the team can execute the task at hand.</p> <p>Please do not hesitate to reach out, even if it is for high level guidance. We can try and help. You can email us at <a href="info@networktocode.com">info@networktocode.com</a> and get community input on the <a href="slack.networktocode.com">Network to Code Slack Workspace</a> too.</p> <p>-Jason</p>Jason EdelmanBy now, COVID-19 has impacted all of us in some way, shape, or form. We truly hope everyone is following the guidelines and recommendations from WHO and their local authorities.Automation Principles - Atomicity2020-03-10T00:00:00+00:002020-03-10T00:00:00+00:00https://blog.networktocode.com/post/Principle-Series-Atomicity<p>This is part of a <a href="../Network-Automation-Principles/">series of posts</a> focused on Network Automation Principles.</p> <h2 id="atomicity-in-computer-science">Atomicity in Computer Science</h2> <p>In simple terms, an atomic operation guarantees that an action succeeds completely or fails completely. Going a step deeper, <a href="https://bigdata-guide.blogspot.com/2014/01/what-is-atomicity.html">the included quote</a> provides context and accuracy to describe the principle, in proper terminology:</p> <blockquote> <p>In concurrent programming, an operation (or set of operations) is atomic, linearizable, indivisible or uninterruptible if it appears to the rest of the system to occur instantaneously.</p> </blockquote> <blockquote> <p>Atomicity is a guarantee of isolation from concurrent processes. Additionally, atomic operations commonly have a succeed-or-fail definition — they either successfully change the state of the system, or have no apparent effect.</p> </blockquote> <blockquote> <p>Atomicity is commonly enforced by mutual exclusion, whether at the hardware level building on a cache coherency protocol, or the software level using semaphores or locks. Thus, an atomic operation does not actually occur instantaneously.</p> </blockquote> <p>In computer science, atomicity is often associated with databases and represents the “A” in <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>, which describes a set of properties for database transactions.</p> <h2 id="example">Example</h2> <p>A common example used to illustrate the effect of the principle is transferring money from one account to another. When transferring $100 from Bob’s to Sally’s bank account, it would be disastrous if money was removed from Bob’s account but not deposited to Sally’s account for Bob and Sally, as it would result in a missing $100 to Sally. The transaction would be equally problematic if the money was added to Sally’s account without being removed from Bob’s account for the bank.</p> <h2 id="achieving-atomicity">Achieving Atomicity</h2> <p>There are several mechanisms to achieve atomicity, but for simplicity, the focus will be on locking. A simplified view of this can be seen as:</p> <ul> <li>Identify memory locations that will be accessed.</li> <li>Create locks on each location.</li> <li>Perform a linear set of transactions and save to a new memory location.</li> <li>Wait for all transactions to complete.</li> <li>In a single transaction, swap the memory locations.</li> </ul> <p>This strategy would likely result in the inability for the operation to be multi-threaded, but as alluded to, there are other methods.</p> <h2 id="atomic-operations-in-python">Atomic Operations in Python</h2> <p>In researching examples, <a href="https://blog.qqrs.us/blog/2016/05/01/which-python-operations-are-atomic/">one blog</a> referred to two key explanations. The official <a href="https://docs.python.org/2/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe">Python documentation</a> provides insight into what operations are/are not atomic. Even more to the point is <a href="https://google.github.io/styleguide/pyguide.html#Threading">Google’s styling guide on Threading</a> which provides the below insight.</p> <blockquote> <p>While Python’s built-in data types such as dictionaries appear to have atomic operations, there are corner cases where they aren’t atomic (e.g. if <strong>hash</strong> or <strong>eq</strong> are implemented as Python methods) and their atomicity should not be relied upon. Neither should you rely on atomic variable assignment (since this in turn depends on dictionaries).</p> </blockquote> <p>Rather interesting to note that even variable assignment are not guaranteed to be safe and atomic.</p> <h2 id="atomic-operation-in-networking">Atomic Operation in Networking</h2> <p>Using a traditional Cisco IOS CLI, applying configurations in a linear fashion can be problematic. Take, for example, creating a VTY ACL. Adding a single line to the new ACL, can enforce an implicit deny and could potentially lock the the operator out. Naturally, the ability to apply configurations atomically has obvious benefits.</p> <p>Luckily, since at least 2007, there has been a feature called “<a href="https://learning.nil.com/assets/Tips-/Replacing-Configuration-on-a-Working-Router.pdf">configure replace</a>”, to replace an entire Cisco IOS configuration. Even better, this is used within the <a href="https://github.com/napalm-automation/napalm/">NAPALM</a> Python library. In order to illustrate the effect, the first attempt will fail to push configurations and demonstrate a proper rollback, and the second attempt will work as intended.</p> <p>Initially, the “intended” configuration will contain this contrived incorrect configuration.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vlan 499 name printers ! vlan 5000 name users </code></pre></div></div> <p>The mistyped VLAN 5000 instead of 500 will be problematic. Let’s observe how NAPALM reacts.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">getpass</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">napalm</span> <span class="kn">import</span> <span class="n">get_network_driver</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">driver</span> <span class="o">=</span> <span class="n">get_network_driver</span><span class="p">(</span><span class="s">'ios'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">hostname</span> <span class="o">=</span> <span class="s">'ios-sw1'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">username</span> <span class="o">=</span> <span class="s">'ntc'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">password</span> <span class="o">=</span> <span class="n">getpass</span><span class="o">.</span><span class="n">getpass</span><span class="p">()</span> <span class="n">Password</span><span class="p">:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span> <span class="o">=</span> <span class="n">driver</span><span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="nb">open</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">load_replace_candidate</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s">'ios-sw1'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">compare_config</span><span class="p">()</span> <span class="s">'+vlan 499</span><span class="se">\n</span><span class="s"> +name printers</span><span class="se">\n</span><span class="s">+vlan 5000</span><span class="se">\n</span><span class="s"> +name users'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">commit_config</span><span class="p">()</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s">"&lt;stdin&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">File</span> <span class="s">"/usr/local/lib/python3.5/dist-packages/napalm/ios/ios.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">533</span><span class="p">,</span> <span class="ow">in</span> <span class="n">commit_config</span> <span class="k">raise</span> <span class="n">ReplaceConfigException</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span> <span class="n">napalm</span><span class="o">.</span><span class="n">base</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">ReplaceConfigException</span><span class="p">:</span> <span class="n">Candidate</span> <span class="n">config</span> <span class="n">could</span> <span class="ow">not</span> <span class="n">be</span> <span class="n">applied</span> <span class="n">Command</span> <span class="n">rejected</span><span class="p">:</span> <span class="n">Bad</span> <span class="n">VLAN</span> <span class="nb">list</span> <span class="o">-</span> <span class="n">character</span> <span class="c1">#5 (EOL) delimits a VLAN </span><span class="n">number</span> <span class="n">which</span> <span class="ow">is</span> <span class="n">out</span> <span class="n">of</span> <span class="n">the</span> <span class="nb">range</span> <span class="mf">1..4094</span><span class="o">.</span> <span class="n">Failed</span> <span class="n">to</span> <span class="nb">apply</span> <span class="n">command</span> <span class="n">vlan</span> <span class="mi">5000</span> <span class="n">Aborting</span> <span class="n">Rollback</span><span class="o">.</span> <span class="n">Rollback</span> <span class="n">failed</span><span class="o">.</span><span class="n">Reverting</span> <span class="n">back</span> <span class="n">to</span> <span class="n">the</span> <span class="n">original</span> <span class="n">configuration</span><span class="p">:</span> <span class="n">flash</span><span class="p">:</span><span class="o">-</span><span class="n">Dec</span><span class="o">-</span><span class="mi">14</span><span class="o">-</span><span class="mi">13</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mf">59.491</span><span class="o">-</span><span class="mi">0</span> <span class="o">...</span> <span class="n">Total</span> <span class="n">number</span> <span class="n">of</span> <span class="n">passes</span><span class="p">:</span> <span class="mi">1</span> <span class="n">Rollback</span> <span class="n">Done</span> <span class="n">The</span> <span class="n">original</span> <span class="n">configuration</span> <span class="n">has</span> <span class="n">been</span> <span class="n">successfully</span> <span class="n">restored</span><span class="o">.</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>As you can see, the comparison of the configuration indicates an attempt to add VLANs 499 and 5000. However, when a configuration commit is attempted, the configuration aborts with the following message <code class="highlighter-rouge">Command rejected: Bad VLAN list - character #5 (EOL) delimits a VLAN number which is out of the range 1..4094</code>. The failure forces a rollback and ensures that both VLANs did not get created.</p> <p>Updating the VLAN 5000 to the correct VLAN 500 and rerunning the process results in a successful operation.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">device</span> <span class="o">=</span> <span class="n">driver</span><span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="nb">open</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">load_replace_candidate</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s">'ios-sw1'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">compare_config</span><span class="p">()</span> <span class="s">'+vlan 499</span><span class="se">\n</span><span class="s"> +name printers</span><span class="se">\n</span><span class="s">+vlan 500</span><span class="se">\n</span><span class="s"> +name users'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">device</span><span class="o">.</span><span class="n">commit_config</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <p>This can be further verified on the switch, by viewing the configuration.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vlan 499 name printers ! vlan 500 name users </code></pre></div></div> <p>While the “configure replace” process likely lacks the integrity that a PostgreSQL or MySQL database will have, it is largely effective in creating an atomic operation.</p> <h2 id="conclusion">Conclusion</h2> <p>An atomic operation is generally safer and implies your intent–outside of rare use cases. In databases, this is often handled in transactions and in practice creating your own atomic actions often requires locks. Perhaps Yoda can best describe atomicity succinctly with his quote, “do or do not. There is no try.”</p> <p><img src="../../../static/images/blog_posts/atomic/yoda.jpg" alt="Yoda Quote" /></p> <p>-Ken</p>Ken CelenzaThis is part of a series of posts focused on Network Automation Principles.Using Ansible through a Bastion Host2020-03-05T00:00:00+00:002020-03-05T00:00:00+00:00https://blog.networktocode.com/post/ansible-bastion-host<p>This blog is going to go over the concepts of using Ansible to interact with a private network through a bastion host. The use of <a href="https://en.wikipedia.org/wiki/Bastion_host">bastion hosts</a> is not new to the industry—they have been used for a while by many companies that need to give users access to private networks. Bastion hosts are typically public facing, hardened systems that work as an entrypoint to systems that are behind a firewall or any other restricted locations.</p> <h2 id="set-up-ssh-public-key-authentication-with-bastion-host">Set up SSH public key authentication with bastion host</h2> <p>When interacting with a bastion host, the recommended first step is to set up SSH public keys and authenticate with the bastion host.</p> <p>If an SSH key doe not already exist, create it by issuing the command <code class="highlighter-rouge">ssh-keygen -t rsa</code></p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# ssh-keygen <span class="nt">-t</span> rsa Generating public/private rsa key pair. Enter file <span class="k">in </span>which to save the key <span class="o">(</span>/root/.ssh/id_rsa<span class="o">)</span>: Created directory <span class="s1">'/root/.ssh'</span><span class="nb">.</span> Enter passphrase <span class="o">(</span>empty <span class="k">for </span>no passphrase<span class="o">)</span>: Enter same passphrase again: Your identification has been saved <span class="k">in</span> /root/.ssh/id_rsa. Your public key has been saved <span class="k">in</span> /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:T5YDoF91QrJk2ZN1b7OYq9Pfn3OWVW518iCJHGIQKeQ root@eb54369adc49 The key<span class="s1">'s randomart image is: +---[RSA 2048]----+ | .. +++++oo . | | ....+++=o . . | | E. .+o + . .o| | . . .o.o =.*| | . S = + *+| | + . . =| | . .. .o| | ... o=| | .. .+*| +----[SHA256]-----+ root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# </span></code></pre></div></div> <p>Since this is just for demonstration, I’ve kept everything as default when creating the keys.</p> <blockquote> <p>Note: <a href="https://kb.iu.edu/d/aews">Click here</a> if you want to know more about how to set up an SSH public key with more details.</p> </blockquote> <p>After creating the public key, the <code class="highlighter-rouge">ssh-copy-id</code> Linux command can be used to transfer the <code class="highlighter-rouge">id_rsa.pub</code> public key to the bastion host.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# ssh-copy-id ntc@bastion /usr/bin/ssh-copy-id: INFO: Source of key<span class="o">(</span>s<span class="o">)</span> to be installed: <span class="s2">"/root/.ssh/id_rsa.pub"</span> /usr/bin/ssh-copy-id: INFO: attempting to log <span class="k">in </span>with the new key<span class="o">(</span>s<span class="o">)</span>, to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key<span class="o">(</span>s<span class="o">)</span> remain to be installed <span class="nt">--</span> <span class="k">if </span>you are prompted now it is to <span class="nb">install </span>the new keys ntc@bastion password: Number of key<span class="o">(</span>s<span class="o">)</span> added: 1 Now try logging into the machine, with: <span class="s2">"ssh 'ntc@bastion'"</span> and check to make sure that only the key<span class="o">(</span>s<span class="o">)</span> you wanted were added. root@eb54369adc49:/ntc# </code></pre></div></div> <p>Finally, attempt to SSH into the bastion host and make sure it works as expected. The workstation should log in through SSH without having to provide a password.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# ssh ntc@bastion Welcome to Ubuntu 16.04.1 LTS <span class="o">(</span>GNU/Linux 4.4.0-59-generic x86_64<span class="o">)</span> <span class="k">*</span> Documentation: https://help.ubuntu.com <span class="k">*</span> Management: https://landscape.canonical.com <span class="k">*</span> Support: https://ubuntu.com/advantage 466 packages can be updated. 321 updates are security updates. New release <span class="s1">'18.04.3 LTS'</span> available. Run <span class="s1">'do-release-upgrade'</span> to upgrade to it. Last login: Mon Jan 13 13:44:58 2020 from 71.52.24.123 ntc@bastion:~<span class="err">$</span> ntc@bastion:~<span class="nv">$ </span><span class="nb">exit logout </span>Connection to bastion closed. root@eb54369adc49:/ntc# </code></pre></div></div> <p>There are many different ways of creating a public key and transferring it to the remote host. This transfer can also be handled via Ansible, as the below playbook shows:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Generate Key</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">local</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Generate an OpenSSH keypair with the default values (4096 bits, rsa)</span> <span class="na">openssh_keypair</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/root/.ssh/id_rsa</span> <span class="na">owner</span><span class="pi">:</span> <span class="s">root</span> <span class="na">group</span><span class="pi">:</span> <span class="s">root</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy public key to remote host</span> <span class="na">shell</span><span class="pi">:</span> <span class="s">ssh-copy-id ntc@bastion</span> </code></pre></div></div> <h2 id="setting-up-ansible-to-interact-with-a-cisco-ios-device-through-a-bastion-host">Setting up Ansible to Interact with a Cisco IOS Device through a Bastion Host</h2> <p>Now that it is possible to authenticate into the bastion host only using SSH keys and not require user/passwords, it’s time to build the Ansible inventory file with the devices that are going to be targeted by Ansible. Before creating the inventory file, notice how the only way to interact with the Cisco <code class="highlighter-rouge">csr1</code> device is through the <strong>bastion</strong> host.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # Topology ┏━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━┓ ┃ Container ┃──────────┃ Bastion host ┃──────────┃ csr1 ┃ ┗━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━┛ </code></pre></div></div> <p>This first test is using the <code class="highlighter-rouge">ping</code> command to ping the <code class="highlighter-rouge">csr1</code> device from the local container. The results should come back with 100% packet loss.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# ping csr1 <span class="nt">-c</span> 5 PING csr1 <span class="o">(</span>54.84.81.49<span class="o">)</span> 56<span class="o">(</span>84<span class="o">)</span> bytes of data. <span class="nt">---</span> csr1 ping statistics <span class="nt">---</span> 5 packets transmitted, 0 received, 100% packet loss, <span class="nb">time </span>4176ms </code></pre></div></div> <p>After using the SSH command to access the bastion host, the ping command is used again to show that there is IP connectivity between the bastion host and the networking device.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# ssh ntc@bastion Welcome to Ubuntu 16.04.1 LTS <span class="o">(</span>GNU/Linux 4.4.0-59-generic x86_64<span class="o">)</span> <span class="k">*</span> Documentation: https://help.ubuntu.com <span class="k">*</span> Management: https://landscape.canonical.com <span class="k">*</span> Support: https://ubuntu.com/advantage 466 packages can be updated. 321 updates are security updates. New release <span class="s1">'18.04.3 LTS'</span> available. Run <span class="s1">'do-release-upgrade'</span> to upgrade to it. Last login: Mon Jan 13 14:38:26 2020 from 71.52.24.123 ntc@bastion:~<span class="err">$</span> ntc@bastion:~<span class="nv">$ </span>ping csr1 <span class="nt">-c</span> 5 PING csr1 <span class="o">(</span>10.0.0.51<span class="o">)</span> 56<span class="o">(</span>84<span class="o">)</span> bytes of data. 64 bytes from csr1 <span class="o">(</span>10.0.0.51<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>2 <span class="nv">ttl</span><span class="o">=</span>255 <span class="nb">time</span><span class="o">=</span>2.34 ms 64 bytes from csr1 <span class="o">(</span>10.0.0.51<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>3 <span class="nv">ttl</span><span class="o">=</span>255 <span class="nb">time</span><span class="o">=</span>2.26 ms 64 bytes from csr1 <span class="o">(</span>10.0.0.51<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>4 <span class="nv">ttl</span><span class="o">=</span>255 <span class="nb">time</span><span class="o">=</span>1.99 ms 64 bytes from csr1 <span class="o">(</span>10.0.0.51<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>5 <span class="nv">ttl</span><span class="o">=</span>255 <span class="nb">time</span><span class="o">=</span>2.23 ms <span class="nt">---</span> csr1 ping statistics <span class="nt">---</span> 5 packets transmitted, 4 received, 20% packet loss, <span class="nb">time </span>4013ms rtt min/avg/max/mdev <span class="o">=</span> 1.994/2.209/2.342/0.138 ms ntc@bastion:~<span class="err">$</span> </code></pre></div></div> <p>Now that you can verify direct connectivity from the container to the <code class="highlighter-rouge">csr1</code> device is only possible through the bastion host, the <a href="https://github.com/kennylevinsen/sshmuxd/wiki/ProxyCommand">ProxyCommand</a> can be used to gain access to the Cisco device from the container.</p> <blockquote> <p>Note: ProxyCommand is an SSH builtin command feature to provide support for proxy use cases.</p> </blockquote> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# root@eb54369adc49:/ntc# ssh <span class="nt">-o</span> <span class="nv">ProxyCommand</span><span class="o">=</span><span class="s2">"ssh -W %h:%p ntc@bastion"</span> ntc@csr1 The authenticity of host <span class="s1">'csr1 (&lt;no hostip for proxy command&gt;)'</span> can<span class="s1">'t be established. RSA key fingerprint is SHA256:BaRERnJrQ4vzALKQELc6lIs7Kujbe3UCPffPcAhcRKg. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '</span>csr1<span class="s1">' (RSA) to the list of known hosts. Password: csr1# csr1# </span></code></pre></div></div> <h3 id="playbook">Playbook</h3> <p>For this test the <code class="highlighter-rouge">ios_command</code> and <code class="highlighter-rouge">ios_config</code> will be used. The backup parameter has been enabled to show that the backup files are stored in the local machine and not on the bastion host.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Cisco IOS Playbook</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">ios</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SHOW VERSION</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">show version</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CONFIGURE SNMP COMMUNITY</span> <span class="na">ios_config</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">snmp-server community ntc-blog RW</span> <span class="na">backup</span><span class="pi">:</span> <span class="no">true</span> </code></pre></div></div> <h3 id="inventory">Inventory</h3> <p>For now, only the basic credentials and device type are required to target the device.</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">[all:vars]</span> <span class="py">ansible_user</span><span class="p">=</span><span class="s">ntc</span> <span class="py">ansible_ssh_password</span><span class="p">=</span><span class="s">ntc123</span> <span class="nn">[ios]</span> <span class="err">csr1</span> <span class="nn">[ios:vars]</span> <span class="py">ansible_network_os</span><span class="p">=</span><span class="s">ios</span> </code></pre></div></div> <blockquote> <p>Note: There is a dedicated group vars for IOS only for now, since a Juniper device will be tested later in this blog too.</p> </blockquote> <h2 id="first-execution-of-playbook">First Execution of Playbook</h2> <p>Similar to the previous tests, the local machine should not be able to communicate with the Cisco device, since it does not have direct access to the private network.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:ntc<span class="nv">$ </span>ansible-playbook <span class="nt">-i</span> inventory bastion_playbook.yml PLAY <span class="o">[</span>Cisco IOS Playbook] <span class="k">******************************************************************</span> TASK <span class="o">[</span>SHOW VERSION] <span class="k">************************************************************************</span> fatal: <span class="o">[</span>csr1]: FAILED! <span class="o">=&gt;</span> <span class="o">{</span><span class="s2">"changed"</span>: <span class="nb">false</span>, <span class="s2">"msg"</span>: <span class="s2">"timed out"</span><span class="o">}</span> PLAY RECAP <span class="k">*********************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>0 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>1 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 root@eb54369adc49:ntc<span class="err">$</span> </code></pre></div></div> <h3 id="executing-a-playbook-against-a-cisco-ios-device-through-a-bastion-host">Executing a Playbook against a Cisco IOS Device through a Bastion Host</h3> <p>To be able to communicate with the Cisco device the <code class="highlighter-rouge">ProxyCommand</code> needs to be executed when trying to SSH into the device. To start using the <code class="highlighter-rouge">ProxyCommand</code> with Ansible the <code class="highlighter-rouge">ansible_ssh_common_args</code>, a variable needs to be added to the inventory file.</p> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">[all:vars]</span> <span class="py">ansible_user</span><span class="p">=</span><span class="s">ntc</span> <span class="py">ansible_ssh_password</span><span class="p">=</span><span class="s">ntc123</span> <span class="nn">[ios]</span> <span class="err">csr1</span> <span class="nn">[ios:vars]</span> <span class="py">ansible_network_os</span><span class="p">=</span><span class="s">ios</span> <span class="py">ansible_ssh_common_args</span><span class="p">=</span><span class="s">ProxyCommand="ssh -W %h:%p ntc@bastion"</span> </code></pre></div></div> <p>More details can be found in the <a href="https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#delegate-to-vs-proxycommand">Ansible documentation</a> about how to use the <code class="highlighter-rouge">ansible_ssh_common_args</code> and <code class="highlighter-rouge">ProxyCommand</code>. For now, I’ve added a device to the inventory file with the needed credentials, and the <code class="highlighter-rouge">ProxyCommand</code> used earlier and tested through the Linux terminal.</p> <p>The Ansible playbook can now be executed normally using the standard <code class="highlighter-rouge">ansible-playbook</code> command. The only modification needed to interact with the networking device was enabling SSH keys and using the <code class="highlighter-rouge">ansible_ssh_common_args</code> in the inventory file.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# ansible-playbook <span class="nt">-i</span> inventory bastion_playbook.yml <span class="nt">-v</span> Using /ntc/ansible.cfg as config file PLAY <span class="o">[</span>Cisco IOS Playbook] <span class="k">***********************************************************</span> TASK <span class="o">[</span>SHOW VERSION] <span class="k">***************************************************************</span> ok: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span><span class="s2">"ansible_facts"</span>: <span class="o">{</span><span class="s2">"discovered_interpreter_python"</span>: <span class="s2">"/usr/bin/python"</span><span class="o">}</span> <span class="s2">"changed"</span>: <span class="nb">false</span>, <span class="s2">"stdout"</span>: <span class="o">[</span><span class="s2">"Cisco IOS XE Software, Version 16.06.02</span><span class="se">\n</span><span class="s2">Cisco IOS Software [Everest], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.2, RELEASE SOFTWARE (fc2)</span><span class="se">\n</span><span class="s2">Technical Support: http://www.cisco.com techsupport</span><span class="se">\n</span><span class="s2">Copyright (c) 1986-2017 by Cisco Systems, Inc </span><span class="se">\n</span><span class="s2">Compiled Wed 01-Nov-17 07:24 by mcpre</span><span class="se">\n\n\n</span><span class="s2">Cisco IOS-X software, Copyright (c) 2005-2017 by cisco Systems, Inc </span><span class="se">\n</span><span class="s2">All rights reserved. Certain components of Cisco IOS-XE software are</span><span class="se">\n</span><span class="s2">licensed under the GNU General Public License"</span> ...omitted]]<span class="o">}</span> TASK <span class="o">[</span>CONFIGURE SNMP COMMUNITY] <span class="k">***********************************************************</span> changed: <span class="o">[</span>csr1] <span class="o">=&gt;</span> <span class="o">{</span><span class="s2">"backup_path"</span>: <span class="s2">"/ntc/backup/csr1_config.2020-01-13@22:02:16"</span>, <span class="s2">"banners"</span>: <span class="o">{}</span>, <span class="s2">"changed"</span>: <span class="nb">true</span>, <span class="s2">"commands"</span>: <span class="o">[</span><span class="s2">"snmp-server community ntc-blog RW"</span><span class="o">]</span>, <span class="s2">"date"</span>: <span class="s2">"2020-01-13"</span>, <span class="s2">"filename"</span>: <span class="s2">"csr1_config.2020-01-13@22:02:16"</span>, <span class="s2">"shortname"</span>: <span class="s2">"/ntc backup/csr1_config"</span>, <span class="s2">"time"</span>: <span class="s2">"22:02:16"</span>, <span class="s2">"updates"</span>: <span class="o">[</span><span class="s2">"snmp-server community ntc-blog RW"</span><span class="o">]}</span> PLAY RECAP <span class="k">*********************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>2 <span class="nv">changed</span><span class="o">=</span>1 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <h2 id="setting-up-ansible-to-interact-with-a-juniper-vmx-device-using-netconf-through-a-bastion-host">Setting up Ansible to Interact with a Juniper vMX Device using NETCONF through a Bastion Host</h2> <p>To interact with a Juniper device via a bastion host that uses NETCONF, a different variable is required and a configuration file needs to be built. This <a href="https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#enabling-jump-host-setting">link</a> gives some details on different ways to enable the feature, but for this test I’ll be adding the <code class="highlighter-rouge">ansible_netconf_ssh_config=./ntc/netconf_proxy_config.cfg</code> variable with the path to my <code class="highlighter-rouge">netconf_proxy_config.cfg</code> file.</p> <blockquote> <p>Note: Even thought the Ansible documentation is not entirely clear why <code class="highlighter-rouge">ansible_ssh_common_args</code> does not work with NETCONF devices, maybe this <a href="https://github.com/ansible/ansible/pull/42271#issuecomment-412406156">link</a> can provide insight to that question.</p> </blockquote> <div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">[all:vars]</span> <span class="py">ansible_user</span><span class="p">=</span><span class="s">ntc</span> <span class="py">ansible_ssh_password</span><span class="p">=</span><span class="s">ntc123</span> <span class="nn">[ios]</span> <span class="err">csr1</span> <span class="nn">[ios:vars]</span> <span class="py">ansible_network_os</span><span class="p">=</span><span class="s">ios</span> <span class="py">ansible_ssh_common_args</span><span class="p">=</span><span class="s">ProxyCommand="ssh -W %h:%p ntc@bastion"</span> <span class="nn">[vmx]</span> <span class="err">vmx1</span> <span class="nn">[vmx:vars]</span> <span class="py">ansible_network_os</span><span class="p">=</span><span class="s">junos</span> <span class="py">ansible_netconf_ssh_config</span><span class="p">=</span><span class="s">./ntc/netconf_proxy_config.cfg</span> </code></pre></div></div> <h3 id="setting-up-the-netconf_proxy_configcfg--and-ansiblecfg-file">Setting up the netconf_proxy_config.cfg and ansible.cfg file</h3> <p>The <a href="https://docs.ansible.com/ansible/latest/network/user_guide/network_debug_troubleshooting.html#example-ssh-config-file-ssh-config">Ansible documentation</a> shows how to build a proper <code class="highlighter-rouge">netconf_proxy_config.cfg</code> file. However, since I’m only testing with a single device, the only requirement is the <code class="highlighter-rouge">ProxyCommand</code> that will establish the SSH connection with the bastion host and the Juniper device.</p> <pre><code class="language-cfg">ProxyCommand ssh -W %h:%p ntc@bastion </code></pre> <p>The next step will be to add a second play using the correct modules to interact with a Juniper device.</p> <h3 id="executing-a-playbook-against-a-juniper-vmx-device-using-netconf-through-a-bastion-host">Executing a Playbook against a Juniper vMX Device using NETCONF through a Bastion Host</h3> <p>The example playbook has been extended to include not only the original Cisco IOS tasks, but an additional play to run the equivalent on Juniper devices. The goal is to run a <code class="highlighter-rouge">show version</code> and make a configuration change to add an SNMP community to the device configuration.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Cisco IOS Playbook</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">ios</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tags</span><span class="pi">:</span> <span class="s">cisco</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SHOW VERSION</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">show version</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CONFIGURE SNMP COMMUNITY</span> <span class="na">ios_config</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">snmp-server community ntc-blog RW</span> <span class="na">backup</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Juniper vMX Playbook</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">vmx</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">netconf</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tags</span><span class="pi">:</span> <span class="s">juniper</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SHOW VERSION</span> <span class="na">junos_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">show version</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CONFIGURE SNMP COMMUNITY</span> <span class="na">junos_config</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="s">set snmp community ntc-blog authorization read-write</span> </code></pre></div></div> <p>After building the playbook, it’s time to execute it and see the results. In this case I ran the playbook and added a <code class="highlighter-rouge">tag</code> so only the Juniper play gets executed.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> root@eb54369adc49:/ntc# ansible-playbook <span class="nt">-i</span> inventory bastion_playbook.yml <span class="nt">--tags</span> juniper <span class="nt">-v</span> Using /ntc/ansible.cfg as config file PLAY <span class="o">[</span>Cisco IOS Playbook] <span class="k">*******************************************************************</span> PLAY <span class="o">[</span>Juniper IOS Playbook] <span class="k">*****************************************************************</span> TASK <span class="o">[</span>SHOW VERSION] <span class="k">*************************************************************************</span> ok: <span class="o">[</span>vmx1] <span class="o">=&gt;</span> <span class="o">{</span><span class="s2">"changed"</span>: <span class="nb">false</span>, <span class="s2">"stdout"</span>: <span class="o">[</span><span class="s2">"Hostname: vmx1</span><span class="se">\n</span><span class="s2">Model: vmx</span><span class="se">\n</span><span class="s2">Junos: 15.1F4.15 </span><span class="se">\n</span><span class="s2">JUNOS Base OS boot [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS Base OS Software Suite [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS Crypto Software Suite [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS OnlineDocumentation [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS 64-bit Kernel Software Suite [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS Routing Software Suite [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS Runtime Software Suite [15.1F4. 15] </span><span class="se">\n</span><span class="s2">JUNOS 64-bit Runtime Software Suite [15.1F4.15] </span><span class="se">\n</span><span class="s2">JUNOS Services AACL PIC package [15.1F4.15] </span><span class="se">\n</span><span class="s2">JUNOS Services Application Level Gateway (xlp64) [15.1F4.15]</span><span class="se">\n</span><span class="s2">JUNOS Services ..omitted"</span><span class="o">]]}</span> TASK <span class="o">[</span>CONFIGURE SNMP COMMUNITY] <span class="k">*************************************************************</span> changed: <span class="o">[</span>vmx1] <span class="o">=&gt;</span> <span class="o">{</span><span class="s2">"changed"</span>: <span class="nb">true</span><span class="o">}</span> PLAY RECAP <span class="k">**********************************************************************************</span> vmx1 : <span class="nv">ok</span><span class="o">=</span>2 <span class="nv">changed</span><span class="o">=</span>1 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <h2 id="summary">Summary</h2> <p>In order to use Ansible through a bastion host with core modules, SSH keys are recommended. When interacting with devices that support NETCONF, the <code class="highlighter-rouge">ansible_netconf_ssh_config</code> variable is required to be enabled and for traditional SSH devices the <code class="highlighter-rouge">ansible_ssh_common_args</code> is required.</p> <p>Other third party modules like <code class="highlighter-rouge">ntc_show_command</code> or NAPALM can also be used but will not be able to support <code class="highlighter-rouge">ansible_ssh_common_args</code> or <code class="highlighter-rouge">ansible_netconf_ssh_config</code>. In this case, the <code class="highlighter-rouge">delegate_to</code> argument will be needed to use a third party module with a jump-host/bastion-host.</p> <p>-Hector</p>Hector IsazaThis blog is going to go over the concepts of using Ansible to interact with a private network through a bastion host. The use of bastion hosts is not new to the industry—they have been used for a while by many companies that need to give users access to private networks. Bastion hosts are typically public facing, hardened systems that work as an entrypoint to systems that are behind a firewall or any other restricted locations.Python Objects2020-02-25T00:00:00+00:002020-02-25T00:00:00+00:00https://blog.networktocode.com/post/Python-Objects<p>“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the <code class="highlighter-rouge">str</code> class.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Creating a string variable </span><span class="o">&gt;&gt;&gt;</span> <span class="n">device_type</span> <span class="o">=</span> <span class="s">"switch"</span> <span class="c1"># device_type is an instance of the str class </span><span class="o">&gt;&gt;&gt;</span> <span class="nb">type</span><span class="p">(</span><span class="n">device_type</span><span class="p">)</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">str</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt;</span><span class="err"> </span></code></pre></div></div> <p>Classes themselves are instances of <code class="highlighter-rouge">type</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Creating a class object normally </span><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Cisco</span><span class="p">:</span> <span class="o">...</span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"cisco"</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span> <span class="o">&lt;</span><span class="k">class</span> <span class="err">'</span><span class="nc">__main__</span><span class="o">.</span><span class="n">Cisco</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Cisco.vendor</span><span class="err"> </span><span class="s">'</span><span class="n">cisco</span><span class="s">'</span><span class="err"> </span><span class="s">&gt;&gt;&gt; isinstance(Cisco, type)</span><span class="err"> </span><span class="s">True</span><span class="err"> </span><span class="s"># Creating a class object explicitly from type</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista = type("Arista", (), {"vendor": "arista"})</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista</span><span class="err"> </span><span class="s">&lt;class '</span><span class="n">__main__</span><span class="o">.</span><span class="n">Arista</span><span class="s">'&gt;</span><span class="err"> </span><span class="s">&gt;&gt;&gt; Arista.vendor</span><span class="err"> </span><span class="s">'</span><span class="n">arista</span><span class="s">'</span><span class="err"> </span><span class="s">&gt;&gt;&gt;</span><span class="err"> </span></code></pre></div></div> <p>In addition to being an instance of a class, objects consist of attributes defining state and behavior. It is common to refer to state attributes as <strong>data attributes</strong> or <strong>variables</strong>, and behaviors are called <strong>methods</strong>. The Python builtin classes generally have many methods, but do not have data attributes. The <em>str</em> class provides methods for:</p> <ul> <li>Manipulating the string (capitalize, format, etc.)</li> <li>Finding information about the string (startswith, isnumeric, etc.)</li> <li>Segmenting the string (partition, split, etc.)</li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span> <span class="o">=</span> <span class="s">"this is a string"</span> <span class="c1"># Capitalize the first letter of each word </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">title</span><span class="p">()</span> <span class="s">'This Is A String'</span> <span class="c1"># Count the number of times the letter i appears </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="s">"i"</span><span class="p">)</span> <span class="mi">3</span> <span class="c1"># Convert the string into a list of words </span><span class="o">&gt;&gt;&gt;</span> <span class="n">example</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="p">[</span><span class="s">"this"</span><span class="p">,</span> <span class="s">"is"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="s">"string"</span><span class="p">]</span> </code></pre></div></div> <p>It is more common for custom classes to have data attributes that store information about the objects they imitate. For an object representing a switch, some example data attributes might be: <em>modules</em>, <em>uptime</em>, and <em>reachability</em>. Some example methods might be: <em>reboot</em>, and <em>ping</em>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">switch</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="n">hostname</span><span class="o">=</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="c1"># Example of a data attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">switch</span><span class="o">.</span><span class="n">uptime</span> <span class="s">'4 years, 3 days'</span> <span class="c1"># Example of a method attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">ping_stats</span> <span class="o">=</span> <span class="n">switch</span><span class="o">.</span><span class="n">ping</span><span class="p">(</span><span class="s">"10.1.1.1"</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="s">'pinging 10.1.1.1 times 5'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ping_stats</span> <span class="p">{</span> <span class="s">'attempts'</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s">'success'</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s">'fail'</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="s">'messages'</span><span class="p">:</span> <span class="p">[</span><span class="s">'pinged 10.1.1.1 in 0.21ms'</span><span class="p">,</span> <span class="s">'pinged 10.1.1.1 in 0.14ms'</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> <span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="data-attributes">Data Attributes</h2> <p>Data attributes are distinguished between class variables and instance variables. Class variables assign values that are the same for all objects of the class. Instance variables assign values for an instance of the class. Class variables are generally used for metadata fields, such as <code class="highlighter-rouge">vendor</code> in the first example. Instance variables contain more interesting information that the object methods act upon. Using the previous example, the <code class="highlighter-rouge">reachability</code> attribute might be used in a report to identify all switches that are unreachable. The Cisco class above is redefined to create instance variables upon initialization for hostname, connection status, and reachability information.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Cisco</span><span class="p">:</span> <span class="c1"># Class variable </span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"cisco"</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hostname</span><span class="p">):</span> <span class="c1"># Instance variables </span> <span class="bp">self</span><span class="o">.</span><span class="n">hostname</span> <span class="o">=</span> <span class="n">hostname</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">ping_reachability</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ping</span><span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="k">if</span> <span class="n">ping_reachability</span><span class="p">[</span><span class="s">"success"</span><span class="p">]:</span> <span class="bp">self</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"authorized"</span> <span class="k">else</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"unauthorized"</span> <span class="k">else</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">reachability</span> <span class="o">=</span> <span class="s">"unreachable"</span> <span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="bp">self</span><span class="o">.</span><span class="n">connected</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">connected</span> <span class="k">else</span> <span class="bp">False</span> <span class="k">def</span> <span class="nf">ping</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">destination</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="p">{</span> <span class="s">"attempts"</span><span class="p">:</span> <span class="n">count</span><span class="p">,</span> <span class="s">"success"</span><span class="p">:</span> <span class="n">success_count</span><span class="p">,</span> <span class="s">"fail"</span><span class="p">:</span> <span class="n">fail_count</span><span class="p">,</span> <span class="s">"messages"</span><span class="p">:</span> <span class="p">[</span><span class="n">message</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">ping_results</span><span class="p">],</span> <span class="p">}</span> </code></pre></div></div> <p>Python uses dictionaries to store class and instance variables. Class variables share the same dictionary across all instances of the class, and instance variables are stored in a unique dictionary per instance. The class dict is stored in <code class="highlighter-rouge">&lt;class&gt;.__dict__</code>, and this is referenced on an instance with <code class="highlighter-rouge">&lt;obj&gt;.__class__.__dict__</code>. The instance dict is stored in <code class="highlighter-rouge">&lt;obj&gt;.__dict__</code>. Here are some examples showing how these two dictionaries behave.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># View the class dict </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">__dict__</span> <span class="n">mappingproxy</span><span class="p">({</span><span class="s">'vendor'</span><span class="p">:</span> <span class="s">'cisco'</span><span class="p">,</span> <span class="o">...</span><span class="p">})</span> <span class="c1"># Create two objects of the Cisco class </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="c1"># Viewing the instance dict </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">}</span> <span class="c1"># Reachability is an instance variable and can have different values across objects </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">reachability</span> <span class="s">'authorized'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">reachability</span> <span class="s">'unreachable'</span> <span class="c1"># The class dict is referenced within instances </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">__dict__</span> <span class="o">==</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__dict__</span> <span class="bp">True</span> <span class="c1"># Changing the vendor on the class is reflected on all instances </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Cisco</span><span class="o">.</span><span class="n">vendor</span> <span class="o">=</span> <span class="s">"CSCO"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'CSCO'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'CSCO'</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="attribute-precedence">Attribute Precedence</h2> <p>This distinction between class and instance variables is important for understanding Python’s attribute access behavior. The standard attribute lookup uses the following precedence until the attribute is found, or an Error is raised:</p> <ol> <li>Attributes defined on the instance</li> <li>Attributes defined on the class</li> <li>Attributes defined on inherited classes</li> </ol> <p>Practically, this means that overriding a class attribute on an instance will only affect the single instance and not other instances.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Cisco</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="c1"># rt1 and rt2 both have a value of "cisco" for the vendor attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="o">==</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="o">==</span> <span class="s">"cisco"</span> <span class="bp">True</span> <span class="c1"># rt1's instance dict does not have an attribute for `vendor` </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="o">=</span> <span class="s">"Cisco"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'Cisco'</span> <span class="c1"># rt1's instance dict was updated with a `vendor` attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">__dict__</span> <span class="p">{</span><span class="s">'hostname'</span><span class="p">:</span> <span class="s">'nyc-hq-rt1'</span><span class="p">,</span> <span class="s">'connected'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="s">'reachability'</span><span class="p">:</span> <span class="s">'authorized'</span><span class="p">,</span> <span class="s">'vendor'</span><span class="p">:</span> <span class="s">'Cisco'</span><span class="p">}</span> <span class="c1"># rt2's vendor attribute remains unchanged </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">vendor</span> <span class="s">'cisco'</span> </code></pre></div></div> <h2 id="mutable-attributes">Mutable Attributes</h2> <p>The above demonstrates the built-in safety provided when naming instance variables collides with class variables. However, this behavior might have surprising results when class variables point to mutable objects (such as lists). The Arista class is rewritten below to have a class variable defining the modules, with the value using a list for modular chassis.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Arista</span><span class="p">:</span> <span class="n">vendor</span> <span class="o">=</span> <span class="s">"arista"</span> <span class="n">modules</span> <span class="o">=</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">]</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="s">"nyc-hq-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span> <span class="o">=</span> <span class="n">Arista</span><span class="p">(</span><span class="s">"nyc-dc-rt1"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">]</span> <span class="c1"># Appending to rt2's `modules` attribute affects rt1 </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">"7500E-72S"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">]</span> <span class="c1"># Using the `+=` operator on rt2's `modules` attributed affects rt1 </span><span class="o">&gt;&gt;&gt;</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span> <span class="o">+=</span> <span class="p">[</span><span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">,</span> <span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="c1"># Both modifications to rt2's `modules` attribute affect the class attribute </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Arista</span><span class="o">.</span><span class="n">modules</span> <span class="p">[</span><span class="s">"7500E-48S"</span><span class="p">,</span> <span class="s">"DCS-7500E-SUP"</span><span class="p">,</span> <span class="s">"7500E-72S"</span><span class="p">,</span> <span class="s">"7500E-36Q"</span><span class="p">]</span> <span class="c1"># All references to `modules` are the same shared object </span><span class="o">&gt;&gt;&gt;</span> <span class="n">Arista</span><span class="o">.</span><span class="n">modules</span> <span class="ow">is</span> <span class="n">rt1</span><span class="o">.</span><span class="n">modulues</span> <span class="ow">is</span> <span class="n">rt2</span><span class="o">.</span><span class="n">modules</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> </code></pre></div></div> <h2 id="conclusion">Conclusion</h2> <p>Understanding how objects work is critical to working with Python. Fundamental to working with objects is managing and using data attributes. Data attributes can be created on the class, or on each instance of the class. Instance level attributes have a higher precedence than class level attributes. Finally, when creating attributes on a class, it is important to consider how they will be used, and that mutable attributes can lead to unexpected behavior.</p> <p>-Jacob</p>Jacob McGill“Everything in Python is an object” has been a common refrain among the Python community. What is an object? A simple definition is: An object is an instance of a class. For example, strings are instances of the str class.Backing up device configurations using Nornir and Ansible2020-02-20T00:00:00+00:002020-02-20T00:00:00+00:00https://blog.networktocode.com/post/nornir_ansible_backup<p>In this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but <strong>it depends :)</strong>. There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.</p> <p>Hopefully, the examples shown here can help you determine which tool best fits the need. I’m not going deep into the details on the different technologies or explaining all the components of each of the files, but I’ll provide some links to learn more about them if needed.</p> <h2 id="installation">Installation</h2> <p>If you want to follow along with the blog, Nornir and Ansible need to be installed in the machine locally or you will need to use tools that isolate dependencies like Docker and virtual environments. I ended up using Docker so if you have Docker installed in your machine and know how to use it, then feel free to use the <code class="highlighter-rouge">Dockerfile</code> below:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM python:3.7-stretch WORKDIR /ntc RUN apt-get update \ &amp;&amp; apt-get install tree RUN pip install black \ ansible \ nornir </code></pre></div></div> <p>After adding and saving the commands to the <code class="highlighter-rouge">Dockerfile</code> run the following commands to build the container image and start running the container.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build -t nornir . docker run -it --name nornir_ansible -v ${PWD}:/ntc nornir_ansible:latest /bin/bash </code></pre></div></div> <h2 id="backing-up-with-nornir">Backing up with Nornir</h2> <p><a href="https://nornir.readthedocs.io/en/latest/#welcome-to-nornir-s-documentation">Nornir</a> is an automation tool used to interact with networking devices. This tool is a bit different compared to other tools in the way that you write your own Python code to control the automation. For example, Ansible is written in Python but uses its own DSL which you use to describe what you want to have done. Nornir does not need a DSL instead, you can write everything in Python.</p> <h3 id="configuration-file">Configuration File</h3> <p>Now that everything has been installed, the first file to create is the Nornir configuration file. This file is similar to the <code class="highlighter-rouge">ansible.cfg</code> file which helps set all the system defaults, but in this case it’s going to be used to find the inventory file. Click on the following links to learn more about Nornir and Ansible configuration files:</p> <ul> <li><a href="https://nornir.readthedocs.io/en/stable/configuration/index.htmlhttps://nornir.readthedocs.io/en/stable/configuration/index.html">Nornir</a></li> <li><a href="https://docs.ansible.com/ansible/2.4/intro_configuration.html">Ansible</a></li> </ul> <p>Create a file called <code class="highlighter-rouge">nornir_config.yml</code> and inside the file store the following content:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="na">inventory</span><span class="pi">:</span> <span class="na">plugin</span><span class="pi">:</span> <span class="s">nornir.plugins.inventory.ansible.AnsibleInventory</span> <span class="na">options</span><span class="pi">:</span> <span class="na">hostsfile</span><span class="pi">:</span> <span class="s2">"</span><span class="s">./inventory"</span> </code></pre></div></div> <blockquote> <p>Note: Since this container has access to the local machine, any tool or editors can be used with the same files the container is able to see. I’ll be using Visual Studio Code in my local machine.</p> </blockquote> <p>Inside the container run the <code class="highlighter-rouge">cat</code> command to make sure changes took place in the file.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>nornir_config.yml <span class="nt">---</span> inventory: plugin: nornir.plugins.inventory.ansible.AnsibleInventory options: hostsfile: <span class="s2">"./inventory"</span> </code></pre></div></div> <blockquote> <p>Note: The plugin used here <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/ansible.html">AnsibleInventory</a> allows us to source an inventory file that would normally be consumed by Ansible. There are other <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/index.html">plugins</a> that can be used like <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/simple.html">simple</a>, <a href="https://nornir.readthedocs.io/en/latest/plugins/inventory/nsot.html">NSOT</a> and <a href="https://nornir.readthedocs.io/en/stable/plugins/inventory/netbox.html#module-nornir.plugins.inventory.netbox">NetBox</a>.</p> </blockquote> <h3 id="inventory-file">Inventory File</h3> <p>The inventory file built here will be used for both Ansible and Nornir. The main differences here will be the hostvars names. For now Nornir will be using <code class="highlighter-rouge">hostname</code>, <code class="highlighter-rouge">platform</code>, <code class="highlighter-rouge">username</code> and <code class="highlighter-rouge">password</code>. Later, when the Ansible section comes, up a few other variables will be added. Create a file called <code class="highlighter-rouge">inventory</code> and place it in the same directory path as the <code class="highlighter-rouge">nornir_config.yml</code> file. Save it and run the <code class="highlighter-rouge">cat</code> command to make sure it saved properly and it’s seen by the container.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>inventory <span class="o">[</span>all] csr1 <span class="nb">hostname</span><span class="o">=</span>csr1 <span class="nv">platform</span><span class="o">=</span>ios <span class="nv">username</span><span class="o">=</span>ntc_user <span class="nv">password</span><span class="o">=</span>pass123 </code></pre></div></div> <p>Click on the following links to learn more about Nornir and Ansible inventory:</p> <ul> <li><a href="https://nornir.readthedocs.io/en/stable/tutorials/intro/inventory.html#Inventory">Nornir</a></li> <li><a href="https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#how-to-build-your-inventory">Ansible</a></li> </ul> <h3 id="backup-python-script">Backup Python Script</h3> <p>Now that the configuration and inventory file has been created it’s time to build the script that will backup the device configuration. The script will consist of functions that will build the necessary directory, backup files, and two different functions using the <code class="highlighter-rouge">netmiko</code> and <code class="highlighter-rouge">napalm</code> modules to backup the device configurations.</p> <p>The first step to building the script is to import all the needed libraries. In this case, the libraries needed are:</p> <ul> <li><a href="https://docs.python.org/3/library/os.html#module-os"><strong>os</strong></a>: This library is going to be used to help interact with the local environment and build the backup directory and file names for the configurations.</li> <li><a href="https://nornir.readthedocs.io/en/latest/ref/api/nornir.html#core"><strong>nornir</strong></a>: Nornir is an automation framework written in Python to be used with Python. <ul> <li><a href="https://nornir.readthedocs.io/en/latest/ref/api/nornir.html#initnornir"><strong>InitNornir</strong></a>: Object used to provide the configuration file containing system defaults. In this case, it’s being used to provide the inventory file.</li> <li><a href="https://nornir.readthedocs.io/en/latest/plugins/tasks/networking.html#nornir.plugins.tasks.networking.netmiko_send_command"><strong>netmiko_send_command</strong></a>: It’s a netmiko function used to send arbitrary commands to networking devices.</li> <li><a href="https://nornir.readthedocs.io/en/latest/plugins/tasks/networking.html#nornir.plugins.tasks.networking.napalm_get"><strong>napalm_get</strong></a>: Function that comes from the NAPALM library to get <code class="highlighter-rouge">facts</code> from the device. In this case, it will be used to retrieve the device configuration.</li> </ul> </li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span> <span class="kn">import</span> <span class="nn">logging</span> <span class="kn">from</span> <span class="nn">nornir</span> <span class="kn">import</span> <span class="n">InitNornir</span> <span class="kn">from</span> <span class="nn">nornir.plugins.tasks.networking</span> <span class="kn">import</span> <span class="n">netmiko_send_command</span><span class="p">,</span> <span class="n">napalm_get</span> </code></pre></div></div> <p>Two global variables need to be created. The first one, <code class="highlighter-rouge">BACKUP_DIR</code>, is the directory name used to store the backed up files and the second,<code class="highlighter-rouge">nr</code>, is used to carry the data stored in the <code class="highlighter-rouge">nornir_config.yml</code> file. In this case, it will contain the inventory file data.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">BACKUP_DIR</span> <span class="o">=</span> <span class="s">"backups/"</span> <span class="n">nr</span> <span class="o">=</span> <span class="n">InitNornir</span><span class="p">(</span><span class="n">config_file</span><span class="o">=</span><span class="s">"./nornir_config.yml"</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">create_backups_dir</code> is used to create a local directory called <code class="highlighter-rouge">backups</code> and it’s using the <code class="highlighter-rouge">os</code> library to create a directory with the name stored in the <code class="highlighter-rouge">BACKUP_DIR</code> variable.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">create_backups_dir</span><span class="p">():</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">):</span> <span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">save_config_to_file</code> function is used to create the backed up device configuration files inside the <code class="highlighter-rouge">backups</code> directory. Notice the parameters used will take as input <code class="highlighter-rouge">method</code>, <code class="highlighter-rouge">hostname</code> and <code class="highlighter-rouge">config</code>, these parameters will get their data from the <code class="highlighter-rouge">get_netmiko_backups()</code> and <code class="highlighter-rouge">get_napalm_backups()</code> functions that will be created in the next few steps.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">save_config_to_file</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="p">):</span> <span class="n">filename</span> <span class="o">=</span> <span class="n">f</span><span class="s">"{hostname}-{method}.cfg"</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BACKUP_DIR</span><span class="p">,</span> <span class="n">filename</span><span class="p">),</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">config</span><span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">get_netmiko_backups</code> function will run the <code class="highlighter-rouge">netmiko_send_command</code> module to send a <strong>“show run”</strong> command to the network device stored in the <code class="highlighter-rouge">nr</code> variable and it will be stored in the <code class="highlighter-rouge">backups_results</code> variable.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_netmiko_backups</span><span class="p">():</span> <span class="n">backup_results</span> <span class="o">=</span> <span class="n">nr</span><span class="o">.</span><span class="n">run</span><span class="p">(</span> <span class="n">task</span><span class="o">=</span><span class="n">netmiko_send_command</span><span class="p">,</span> <span class="n">command_string</span><span class="o">=</span><span class="s">"show run"</span> <span class="p">)</span> <span class="k">for</span> <span class="n">hostname</span> <span class="ow">in</span> <span class="n">backup_results</span><span class="p">:</span> <span class="n">save_config_to_file</span><span class="p">(</span> <span class="n">method</span><span class="o">=</span><span class="s">"netmiko"</span><span class="p">,</span> <span class="n">hostname</span><span class="o">=</span><span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">backup_results</span><span class="p">[</span><span class="n">hostname</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">result</span><span class="p">,</span> <span class="p">)</span> </code></pre></div></div> <p>The <code class="highlighter-rouge">backups_results</code> variable will end up with a value that stores the hostname and configuration as a result looking something like this:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; print(backup_results) AggregatedResult (netmiko_send_command): {'csr1': MultiResult: [Result: "netmiko_send_command"]} &gt;&gt;&gt; &gt;&gt;&gt; </code></pre></div></div> <blockquote> <p>Note: <code class="highlighter-rouge">AggregatedResult</code> is a dict-like object that aggregates the results for all devices. You can access each individual result by doing <code class="highlighter-rouge">my_aggr_result["hostname_of_device"]</code></p> </blockquote> <p>The data inside <code class="highlighter-rouge">backups_results</code> can be extracted by using a for loop and stored in <code class="highlighter-rouge">hostname</code>. The data iterated through with the for loop will return the hostname information and backup configuration.</p> <p>The <code class="highlighter-rouge">get_napalm_backups</code> function will be built the same way, except this time it will be using the <code class="highlighter-rouge">napalm_get</code> module to extract the device configuration.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_napalm_backups</span><span class="p">():</span> <span class="n">backup_results</span> <span class="o">=</span> <span class="n">nr</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">task</span><span class="o">=</span><span class="n">napalm_get</span><span class="p">,</span> <span class="n">getters</span><span class="o">=</span><span class="p">[</span><span class="s">"config"</span><span class="p">])</span> <span class="k">for</span> <span class="n">hostname</span> <span class="ow">in</span> <span class="n">backup_results</span><span class="p">:</span> <span class="n">config</span> <span class="o">=</span> <span class="n">backup_results</span><span class="p">[</span><span class="n">hostname</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">result</span><span class="p">[</span><span class="s">"config"</span><span class="p">][</span><span class="s">"startup"</span><span class="p">]</span> <span class="n">save_config_to_file</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s">"napalm"</span><span class="p">,</span> <span class="n">hostname</span><span class="o">=</span><span class="n">hostname</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">)</span> </code></pre></div></div> <p>The last function <code class="highlighter-rouge">main()</code> will execute all the functions and the code entry point is added to start the script when executed.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="n">create_backups_dir</span><span class="p">()</span> <span class="n">get_netmiko_backups</span><span class="p">()</span> <span class="n">get_napalm_backups</span><span class="p">()</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </code></pre></div></div> <p>Once the script has been built, execute the command <code class="highlighter-rouge">black</code> to clean up and make sure all formatting is correct and use the command <code class="highlighter-rouge">cat</code> to make sure everything has been saved correctly. Click on the following link to learn more about <a href="https://pypi.org/project/black/"><strong>black</strong></a>.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# black backups.py All <span class="k">done</span><span class="o">!</span> ✨ 🍰 ✨ 1 file left unchanged. </code></pre></div></div> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@6b91330f9674:/ntc# <span class="nb">cat </span>backups.py import os from nornir import InitNornir from nornir.plugins.tasks.networking import netmiko_send_command, napalm_get BACKUP_DIR <span class="o">=</span> <span class="s2">"backups/"</span> nr <span class="o">=</span> InitNornir<span class="o">(</span><span class="nv">config_file</span><span class="o">=</span><span class="s2">"./nornir_config.yml"</span><span class="o">)</span> def create_backups_dir<span class="o">()</span>: <span class="k">if </span>not os.path.exists<span class="o">(</span>BACKUP_DIR<span class="o">)</span>: os.mkdir<span class="o">(</span>BACKUP_DIR<span class="o">)</span> def save_config_to_file<span class="o">(</span>method, <span class="nb">hostname</span>, config<span class="o">)</span>: filename <span class="o">=</span> f<span class="s2">"{hostname}-{method}.cfg"</span> with open<span class="o">(</span>os.path.join<span class="o">(</span>BACKUP_DIR, <span class="nb">hostname</span><span class="o">)</span>, <span class="s2">"w"</span><span class="o">)</span> as f: f.write<span class="o">(</span>config<span class="o">)</span> def get_netmiko_backups<span class="o">()</span>: backup_results <span class="o">=</span> nr.run<span class="o">(</span> <span class="nv">task</span><span class="o">=</span>netmiko_send_command, <span class="nv">command_string</span><span class="o">=</span><span class="s2">"show run"</span> <span class="o">)</span> <span class="k">for </span><span class="nb">hostname </span><span class="k">in </span>backup_results: save_config_to_file<span class="o">(</span> <span class="nv">method</span><span class="o">=</span><span class="s2">"netmiko"</span>, <span class="nb">hostname</span><span class="o">=</span><span class="nb">hostname</span>, <span class="nv">config</span><span class="o">=</span>backup_results[hostname][0].result, <span class="o">)</span> def get_napalm_backups<span class="o">()</span>: backup_results <span class="o">=</span> nr.run<span class="o">(</span><span class="nv">task</span><span class="o">=</span>napalm_get, <span class="nv">getters</span><span class="o">=[</span><span class="s2">"config"</span><span class="o">])</span> <span class="k">for </span><span class="nb">hostname </span><span class="k">in </span>backup_results: config <span class="o">=</span> backup_results[hostname][0].result[<span class="s2">"config"</span><span class="o">][</span><span class="s2">"startup"</span><span class="o">]</span> save_config_to_file<span class="o">(</span><span class="nv">method</span><span class="o">=</span><span class="s2">"napalm"</span>, <span class="nb">hostname</span><span class="o">=</span><span class="nb">hostname</span>, <span class="nv">config</span><span class="o">=</span>config<span class="o">)</span> def main<span class="o">()</span>: create_backups_dir<span class="o">()</span> get_netmiko_backups<span class="o">()</span> get_napalm_backups<span class="o">()</span> <span class="k">if </span>__name__ <span class="o">==</span> <span class="s2">"__main__"</span>: main<span class="o">()</span> </code></pre></div></div> <h3 id="running-the-python-script">Running the Python Script</h3> <p>Now that the script is complete, it’s time to execute it by using the <code class="highlighter-rouge">Python</code> command and file name right next to it <code class="highlighter-rouge">python backups.py</code>.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# python backups.py root@c2e446b364a8:/ntc# </code></pre></div></div> <p>The <code class="highlighter-rouge">tree</code> command can be used to view the configuration files stored in the directory <code class="highlighter-rouge">backups</code> configuration files stored inside of it with the specified file names built in the <code class="highlighter-rouge">save_config_to_file</code> function.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── inventory ├── nornir.log └── nornir_config.yml 1 directory, 7 files </code></pre></div></div> <blockquote> <p>Note: Take a look at the backed up configuration files using the <code class="highlighter-rouge">cat</code> command or an editor of choice. They are left off from this as they take up a lot of screen space.</p> </blockquote> <p>By default, Nornir automatically configures logging when InitNornir is called and a file is created locally that can be viewed what functions where ran that belong to the <code class="highlighter-rouge">Nornir</code> library.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>nornir.log 2020-01-02 21:33:26,519 - nornir.core - INFO - run<span class="o">()</span> - Running task <span class="s1">'netmiko_send_command'</span> with args <span class="o">{</span><span class="s1">'command_string'</span>: <span class="s1">'show run'</span><span class="o">}</span> on 1 hosts 2020-01-02 21:33:32,134 - nornir.core - INFO - run<span class="o">()</span> - Running task <span class="s1">'napalm_get'</span> with args <span class="o">{</span><span class="s1">'getters'</span>: <span class="o">[</span><span class="s1">'config'</span><span class="o">]}</span> on 1 hosts </code></pre></div></div> <h2 id="backing-up-with-ansible">Backing up with Ansible</h2> <p>Ansible is a configuration management tool originally created to interact with Linux servers, eventually it started gaining traction in the networking industry to manage networking device configurations. Ansible was built using Python but it uses YAML to configure all the automation tasks. The idea is to simplify the work that it takes to start automating.</p> <h3 id="configuration-file-1">Configuration file</h3> <p>The <code class="highlighter-rouge">ansible.cfg</code> configuration file is used to set some of the system default values. Create the <code class="highlighter-rouge">ansible.cfg</code> file in the root of the directory next to all the other files created previously. Then use the <code class="highlighter-rouge">cat</code> command to make sure everything was stored correctly.</p> <ul> <li><strong>inventory</strong>: Sets the location of the inventory file, by default it will look for it in <code class="highlighter-rouge">etc/ansible/hosts</code> or while running the playbook the <code class="highlighter-rouge">-i</code> flag can be used to point to the location of the file. In this case, it is pointed to the inventory file we will create in the next step within the same directory.</li> <li><strong>host_key_checking</strong>: This has been disabled to prevent host key checking and prevent issues when running the playbook.</li> <li><strong>interpreter_python</strong>: Has been set in the <code class="highlighter-rouge">ansible.cfg</code> file to prevent a new warning added to the new version of Ansible and specify what interpreter to use.</li> <li><strong>gathering</strong>: Is set to smart to gather facts by default, but don’t regather if already gathered.</li> </ul> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>ansible.cfg <span class="o">[</span>defaults] inventory <span class="o">=</span> ./inventory host_key_checking <span class="o">=</span> False interpreter_python <span class="o">=</span> /usr/bin/python gathering <span class="o">=</span> smart </code></pre></div></div> <h3 id="inventory">Inventory</h3> <p>Edit the same inventory file that was used for Nornir, except this time add the following variables for Ansible.</p> <ul> <li><code class="highlighter-rouge">ansible_user=ntc_user</code></li> <li><code class="highlighter-rouge">ansible_password=pass123</code></li> <li><code class="highlighter-rouge">ansible_network_os=ios</code></li> </ul> <p>Again use the <code class="highlighter-rouge">cat</code> command to make sure the changes took effect.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>inventory <span class="o">[</span>all] csr1 <span class="nb">hostname</span><span class="o">=</span>csr1 <span class="nv">platform</span><span class="o">=</span>ios <span class="nv">username</span><span class="o">=</span>ntc_user <span class="nv">password</span><span class="o">=</span>pass123 <span class="nv">ansible_user</span><span class="o">=</span>ntc_user <span class="nv">ansible_password</span><span class="o">=</span>pass123 <span class="nv">ansible_network_os</span><span class="o">=</span>ios </code></pre></div></div> <h3 id="playbook">Playbook</h3> <p>The first part to the Playbook is the play definition. The play definition will contain a list of keys such as:</p> <ul> <li><strong>name</strong>: Any arbitrary name for the Playbook.</li> <li><strong>hosts</strong>: The devices that will be targeted in the scope of the play.</li> <li><strong>connection</strong>: Type of connection. In this case <code class="highlighter-rouge">network_cli</code> which is basically SSH.</li> <li><strong>tasks</strong>: The key that contains all the automation steps that belong to the play.</li> <li><strong>gather_facts</strong>: Before Ansible version 2.9, this key was normally disabled when interacting with networking devices. Now, it has the ability to gather device facts at the play definition level rather than at the task level.</li> </ul> <p>The second part of the Playbook nested under the <code class="highlighter-rouge">tasks</code> key will have a list of <code class="highlighter-rouge">key:value</code> pair tasks.</p> <ul> <li><strong>name</strong>: Optional but recommended key that can have any arbitrary name for the task.</li> <li><a href="https://docs.ansible.com/ansible/latest/modules/ios_config_module.html#ios-config-manage-cisco-ios-configuration-sections"><strong>ios_config</strong></a>: This is the Python module used to run configuration commands against <code class="highlighter-rouge">ios</code> devices.</li> <li><a href="https://docs.ansible.com/ansible/latest/modules/ios_config_module.html#parameters"><strong>backup</strong></a>: Module parameter used to backup the device configuration. By default, if the backup directory is not created then it will create one in the same location as the Ansible playbook and store the configuration files in the backup directory.</li> </ul> <blockquote> <p>Note: A recent parameter was added as <code class="highlighter-rouge">backup_options</code> with sub-options to change the directory path rather than the default which is <code class="highlighter-rouge">backup</code> and the ability to change the filename of the device configuration. In this case, the default value is set with this format <code class="highlighter-rouge">&lt;hostname&gt;_config.&lt;current-date&gt;@&lt;current-time&gt;</code></p> </blockquote> <p>Create the Ansible Playbook and call it <code class="highlighter-rouge">backup_playbook.yml</code>. Use the <code class="highlighter-rouge">cat</code> command to make sure the content is stored correctly:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# <span class="nb">cat </span>backup_playbook.yml <span class="nt">---</span> - name: BACKUP PLAYBOOK hosts: csr1 connection: network_cli gather_facts: <span class="nb">yes </span>tasks: - name: BACKUP USING IOS_CONFIG ios_config: backup: <span class="nb">yes</span> </code></pre></div></div> <h3 id="running-the-playbook">Running the Playbook</h3> <p>To run the Ansible Playbook use the command <code class="highlighter-rouge">ansible-playbook backup_playbook.yml</code>. The output is shown below:</p> <blockquote> <p>Note: There is an extra task that says <code class="highlighter-rouge">Gathering Facts</code> which is a new feature added to Ansible that now gathers facts from networking devices at the play definition level. Take a look at this blog <a href="https://blog.networktocode.com/post/ansible-gather_facts-enabled/">Ansible gather_facts for Networking Devices</a> to learn a little more about it.</p> </blockquote> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml PLAY <span class="o">[</span>BACKUP PLAYBOOK] <span class="k">**********************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">**********************************************************************************</span> <span class="o">[</span>WARNING]: Ignoring <span class="nb">timeout</span><span class="o">(</span>10<span class="o">)</span> <span class="k">for </span>ios_facts ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_CONFIG] <span class="k">***************************************************************************</span> changed: <span class="o">[</span>csr1] PLAY RECAP <span class="k">***********************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>2 <span class="nv">changed</span><span class="o">=</span>1 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>Use the tree command to view the content inside and notice how the device configuration is stored in the <code class="highlighter-rouge">backup</code> directory:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── ansible.cfg ├── backup │   └── csr1_config.2020-01-03@15:43:16 ├── backup_playbook.yml ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── inventory ├── nornir.log └── nornir_config.yml 2 directories, 10 files </code></pre></div></div> <h3 id="backing-up-with-ios_facts">Backing up with ios_facts</h3> <p>There’s another way of backing up the device configurations by using <code class="highlighter-rouge">ios_facts</code>, except this time rather than creating a new task to gather facts we can leverage the gathering of facts that takes place at the play definition level and add a task that will create a new directory to store the device configurations and another task to copy the content from <code class="highlighter-rouge">ansible_facts</code> as a configuration file.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP PLAYBOOK</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">csr1</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">yes</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP USING IOS_CONFIG</span> <span class="na">ios_config</span><span class="pi">:</span> <span class="na">backup</span><span class="pi">:</span> <span class="s">yes</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CREATE BACKUP DIRECTORY</span> <span class="na">file</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">./ansible_backup</span> <span class="na">state</span><span class="pi">:</span> <span class="s">directory</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">BACKUP USING IOS_FACTS</span> <span class="na">copy</span><span class="pi">:</span> <span class="na">content</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_net_config</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">dest</span><span class="pi">:</span> <span class="s">./ansible_backup/{{ inventory_hostname }}_config.{{ now() }}.cfg</span> </code></pre></div></div> <p>In the playbook above, the <code class="highlighter-rouge">copy</code> module is used to copy content stored in the <code class="highlighter-rouge">ansible_net_config</code> key that is found inside <code class="highlighter-rouge">ansible_facts</code>. Any <code class="highlighter-rouge">net_*</code> key under <code class="highlighter-rouge">ansible_facts</code> can be accessed directly by using the <code class="highlighter-rouge">ansible_net_*</code> key rather than accessing the nested data structure. Now the new tasks have been added the playbook can be run again.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# ansible-playbook backup_playbook.yml PLAY <span class="o">[</span>BACKUP PLAYBOOK] <span class="k">**********************************************************************************</span> TASK <span class="o">[</span>Gathering Facts] <span class="k">**********************************************************************************</span> <span class="o">[</span>WARNING]: Ignoring <span class="nb">timeout</span><span class="o">(</span>10<span class="o">)</span> <span class="k">for </span>ios_facts ok: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_CONFIG] <span class="k">***************************************************************************</span> changed: <span class="o">[</span>csr1] TASK <span class="o">[</span>CREATE BACKUP DIRECTORY] <span class="k">****************************************************************************</span> changed: <span class="o">[</span>csr1] TASK <span class="o">[</span>BACKUP USING IOS_FACTS] <span class="k">*****************************************************************************</span> changed: <span class="o">[</span>csr1] PLAY RECAP <span class="k">*************************************************************************************************</span> csr1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>3 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <p>Notice how every time the first task is run a new backup configuration is created since it’s been run with a different time stamp. The same can be done with the <code class="highlighter-rouge">ios_facts</code> backup method.</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@c2e446b364a8:/ntc# tree <span class="nb">.</span> ├── Dockerfile ├── ansible.cfg ├── ansible_backup │   └── csr1_config.2020-01-03 19:01:43.277008.cfg ├── backup │   ├── csr1_config.2020-01-03@15:43:16 │   └── csr1_config.2020-01-03@19:01:42 ├── backup_playbook.yml ├── backups │   ├── csr1-napalm.cfg │   └── csr1-netmiko.cfg ├── backups.py ├── csr1_config.2020-01-03@15:44:07 ├── inventory ├── nornir.log └── nornir_config.yml 3 directories, 13 files </code></pre></div></div> <h3 id="summary">Summary</h3> <p>There are many automation tools and modules that can help automate the backup of device configurations. The ones shown in this blog are some of the basic ways to do it, which ever method you choose will do the job. There can be different modifications that can be done to the Nornir script or even the Ansible playbook to better fit your application and hopefully what I have shown here can give you a start on what tool to use.</p> <p>-Hector</p>Hector IsazaIn this blog, I’m going to go over how to run a configuration backup on a Cisco IOS device using Ansible and Nornir. I’ve heard many people say, “well which tool should I use to run against my infrastructure?” That’s a great question but it depends :). There are many factors that could affect the decision of using a specific tool i.e skill level, time, access to tools, team preference…the list goes on.My first REST API work with Python2020-02-11T00:00:00+00:002020-02-11T00:00:00+00:00https://blog.networktocode.com/post/Basic-API-use-with-python<p>I come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.</p> <p>One of the hardest parts of writing this was choosing an API to discuss. Some of the things I really wanted from an API were:</p> <ul> <li>A Swagger page: I always thought of this as a webpage that has the API laid out nicely and gives the opportunity to run the API calls directly on it. This is more technically known as an Open API spec.</li> <li>A platform that was common in the networking world that people would be likely to run across.</li> <li>A platform that was easily available and didn’t require you to build a new system.</li> </ul> <p>Sadly, this combo doesn’t seem to exist. In the end, I decided to go over 2 APIs:</p> <ul> <li>NetBox <ul> <li>“An open source web application designed to help manage and document computer networks.”</li> <li>Documentation that is easy for a beginner with its Swagger page.</li> <li>https://netbox.readthedocs.io/en/stable/</li> </ul> </li> <li>ACI <ul> <li>A Network policy-based automation model for data centers.</li> <li>Cisco DevNet has an <a href="https://devnetsandbox.cisco.com/RM/Diagram/Index/5a229a7c-95d5-4cfd-a651-5ee9bc1b30e2?diagramType=Topology">always on sandbox</a> where you can access a working ACI environment at any time.</li> <li>When I was working on this API about a year ago, I found the documentation hard to navigate. I can see how it would work well for someone with more experience, but it didn’t work well for me at the time, and I haven’t kept up to date to know if their documentation has gotten more friendly.</li> <li>https://learningnetwork.cisco.com/docs/DOC-32331</li> </ul> </li> </ul> <h2 id="rest-api-core-concepts">REST API Core Concepts</h2> <p>While I’m not going to go to in depth, I want to walk through a few examples to give you the general idea of how a REST API works.</p> <p>There are two main methods of authentication. One where the API is provided a username and password and the system gives back a temporary token. This process is often used in systems where you are running a system such as ACI. In the other method, a token you generate from a UI serves as your authentication. These are often used for web applications such as Slack, Twitter, Facebook, and fortunately NetBox, so we can see how to do that.</p> <p>The general idea behind the REST API is you are going to an URL rather than typing in commands. Let’s look at this like a Cisco CLI. For example, your command might be <code class="highlighter-rouge">show run interface gi1</code>. The equivalent URL might be something like <code class="highlighter-rouge">device.com/api/conf/interface/gi1</code> so to work with <code class="highlighter-rouge">gi2</code> the URL would just change <code class="highlighter-rouge">device.com/api/conf/interface/gi2</code>.</p> <p>The main types of calls are below.</p> <ul> <li><code class="highlighter-rouge">GET</code> <ul> <li>Pulling data.</li> <li>You can’t really cause an issue doing this as long as you don’t pull too much at one time, e.g. pulling 1,000,000 records may bog down the system.</li> <li>Equivalent of a show command.</li> </ul> </li> <li><code class="highlighter-rouge">PUT/POST/PATCH</code> <ul> <li>Changing or creating data.</li> <li>Equivalent of configuration changes.</li> </ul> </li> <li><code class="highlighter-rouge">DELETE</code> <ul> <li>Delete data.</li> </ul> </li> </ul> <p>With the same examples from above, we will do a <code class="highlighter-rouge">GET</code> on <code class="highlighter-rouge">device.com/api/conf/interface/gi1</code>. It returns the dictionary below:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="n">description</span><span class="p">:</span> <span class="s">"Disconnected"</span> <span class="n">status</span><span class="p">:</span> <span class="s">"shut"</span> <span class="n">mode</span><span class="p">:</span> <span class="s">"access"</span> <span class="n">vlans</span><span class="p">:</span> <span class="mi">110</span> <span class="p">}</span> </code></pre></div></div> <p>We now know the description on the interface, the VLAN, and that it is shut down. Let’s say we wanted to make a change, just update it as below and <code class="highlighter-rouge">PUT</code> it to the same URL.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="n">description</span><span class="p">:</span> <span class="s">"VM Server Bob"</span> <span class="n">status</span><span class="p">:</span> <span class="s">"no shut"</span> <span class="n">mode</span><span class="p">:</span> <span class="s">"trunk"</span> <span class="n">vlans</span><span class="p">:</span> <span class="s">"10-100"</span> <span class="p">}</span> </code></pre></div></div> <p>The port is now up, with a new description, as a trunk allowing VLANs 10-100 though. While not quite as easy to understand the url above this URL <code class="highlighter-rouge">https://device.com/restconf/data/Cisco-IOS-XE-native:native/interface=GigabitEthernet/1</code> would work for the RESTCONF API for IOS-XE devices and has an easy to read URL.</p> <h2 id="netbox">NetBox</h2> <p>I am running an instance of NetBox in my lab, the IP is 192.168.0.159, so that’s what all my examples will be of. It’s not hardened or using https, so all of my examples are using http on port 8000. Your box may vary, adjust accordingly.</p> <p>First, let’s build and grab a token from <code class="highlighter-rouge">http://192.168.0.159:8000/user/api-tokens/</code>.</p> <center><img src="../../../static/images/basic_api_use_with_python/Netbox_get_token.jpg" /></center> <p>Next, we’ll look at the swagger documentation page at <code class="highlighter-rouge">http://192.168.0.159:8000/api/docs/</code>. Login to the session</p> <center><img src="../../../static/images/basic_api_use_with_python/authorize_swagger.jpg" /></center> <center><img src="../../../static/images/basic_api_use_with_python/paste_in_token.jpg" /></center> <p>and find an action to do, (e.g. get a device from DCIM,) select that, then select “Try it out.”</p> <center><img src="../../../static/images/basic_api_use_with_python/try_it_out.jpg" /></center> <p>We can pull all devices if we were to look at another option, but the API call gives us lots of options. A screenshot would be overly busy, so I am just going to pull one device in this example. Please note the required tag.</p> <center><img src="../../../static/images/basic_api_use_with_python/execute.jpg" /></center> <p>The API call gives us all the data for that we asked for. Please pay special note to the cURL script. That is the call it made to the NetBox API. It sent a <code class="highlighter-rouge">GET</code> request to <code class="highlighter-rouge">http://192.168.0.159:8000/api/dcim/devices/5/</code> please note the initial line we chose was <code class="highlighter-rouge">/dcim/devices/{id}/</code></p> <center><img src="../../../static/images/basic_api_use_with_python/gives_all_data.jpg" /></center> <p>The cURL script is running an HTTP request to the URL listed, and you could run the same request on nearly all systems. For example, right now you can run “curl http://www.google.com” and make the HTTP call to Google. One of the really nice things about that cURL script is we can go to https://curl.trillworks.com, paste in the cURL script, and it will give us the Python equivalent. NOTE that if the request has the token in it you should be careful. Swap out the token before posting it to the website unless you like giving admin credentials to strangers.</p> <center><img src="../../../static/images/basic_api_use_with_python/gives_python_code.jpg" /></center> <p>Honestly there’s not much to be scared of here. Just working though Swagger and putting a few functions together from what you get from the website. Super easy… Now let’s look at ACI.</p> <h2 id="aci">ACI</h2> <p>In this example, the Cisco DevNet ACI emulator will be used. Set the username, password, and the base URL. This URL will be in every call made. To compare to the NetBox example, the base_url would have been <code class="highlighter-rouge">http://192.168.0.159:8000/api</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">username</span> <span class="o">=</span> <span class="s">"admin"</span> <span class="n">password</span> <span class="o">=</span> <span class="s">"ciscopsdt"</span> <span class="n">base_url</span> <span class="o">=</span> <span class="s">"https://sandboxapicdc.cisco.com/api/"</span> </code></pre></div></div> <p>The login script is honestly one of the most difficult parts of the whole process, as documentation is often sparse and there is no industry standard for field names across REST APIs. The process involves simply logging in, getting a token, and returning it. This is how many APIs work, and there are generally small differences for every API. Most of this was from sample code obtained from DevNet a few years ago.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_token</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">base_url</span><span class="p">):</span> <span class="c1"># Disable warning messages that come with using self signed certs </span> <span class="n">requests</span><span class="o">.</span><span class="n">packages</span><span class="o">.</span><span class="n">urllib3</span><span class="o">.</span><span class="n">disable_warnings</span><span class="p">()</span> <span class="c1"># create credentials structure </span> <span class="n">name_pwd</span> <span class="o">=</span> <span class="p">{</span><span class="s">"aaaUser"</span><span class="p">:</span> <span class="p">{</span><span class="s">"attributes"</span><span class="p">:</span> <span class="p">{</span><span class="s">"name"</span><span class="p">:</span> <span class="n">username</span><span class="p">,</span> <span class="s">"pwd"</span><span class="p">:</span> <span class="n">password</span><span class="p">}}}</span> <span class="n">json_credentials</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">name_pwd</span><span class="p">)</span> <span class="c1"># Create the full URL we need in order to login </span> <span class="c1"># You'll find we do this a lot </span> <span class="n">login_url</span> <span class="o">=</span> <span class="n">base_url</span> <span class="o">+</span> <span class="s">"aaaLogin.json"</span> <span class="c1"># log in to API </span> <span class="c1"># requests is a library that can do HTTP transactions, in this case we POSTing up (Telling it) our </span> <span class="c1"># credentials to get the token back </span> <span class="c1"># Verify=False ignores self signed certs, but would kick off errors if we hadn't disabled them up above </span> <span class="n">post_response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">login_url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json_credentials</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="c1"># getting the token from login response structure </span> <span class="n">auth</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">post_response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="n">login_attributes</span> <span class="o">=</span> <span class="n">auth</span><span class="p">[</span><span class="s">"imdata"</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s">"aaaLogin"</span><span class="p">][</span><span class="s">"attributes"</span><span class="p">]</span> <span class="n">auth_token</span> <span class="o">=</span> <span class="n">login_attributes</span><span class="p">[</span><span class="s">"token"</span><span class="p">]</span> <span class="c1"># create token dictionary that we will use for authentication from here on out for auth cookie </span> <span class="n">cookies</span> <span class="o">=</span> <span class="p">{}</span> <span class="n">cookies</span><span class="p">[</span><span class="s">"APIC-Cookie"</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth_token</span> <span class="k">return</span> <span class="n">cookies</span> </code></pre></div></div> <p>Let’s review the documentation for the ACI API. Once again, this is how it worked back in late 2018, the documentation may have changed since. Access the API inspector,</p> <center><img src="../../../static/images/basic_api_use_with_python/get_api_inspector.jpg" /></center> <p>then proceed to the interesting area in the GUI, paying close attention to your last click. The example will return with the switch information, so I went to <code class="highlighter-rouge">Fabric (top tap) -&gt; Inventory -&gt; POD1 (expanded). Sadly, the URL alone doesn't always give you exactly what you need. In this example I had to cut off the </code>&amp;subscription=yes` section. Often, determinng the URL can be the hardest part.</p> <center><img src="../../../static/images/basic_api_use_with_python/find_call.jpg" /></center> <p>You can also see that the URL is asking for JSON in the <code class="highlighter-rouge">pod-1.json?</code> section. For ACI, it will return either XML or JSON. Personally, I find JSON easier to deal with in Python. Using the JSON library it’s not hard to convert JSON to a dictionary; <code class="highlighter-rouge">https://www.w3schools.com/python/python_json.asp</code> does a better job at explaining the details. Let’s take a look at this process.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span> <span class="k">def</span> <span class="nf">pull_pod1</span><span class="p">(</span><span class="n">cookies</span><span class="p">,</span> <span class="n">base_url</span><span class="p">):</span> <span class="n">sensor_url</span> <span class="o">=</span> <span class="p">(</span> <span class="n">base_url</span> <span class="o">+</span> <span class="s">'/node/mo/topology/pod-1.json?query-target=children&amp;target-subtree-class=fabricNode&amp;query-target-filter=and(not(wcard(fabricNode.dn,</span><span class="si">%22</span><span class="s">__ui_</span><span class="si">%22</span><span class="s">)),and(ne(fabricNode.role,"controller")))'</span> <span class="p">)</span> <span class="n">get_response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">sensor_url</span><span class="p">,</span> <span class="n">cookies</span><span class="o">=</span><span class="n">cookies</span><span class="p">,</span> <span class="n">verify</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="k">return</span> <span class="n">get_response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span> <span class="n">username</span> <span class="o">=</span> <span class="s">"admin"</span> <span class="n">password</span> <span class="o">=</span> <span class="s">"ciscopsdt"</span> <span class="n">base_url</span> <span class="o">=</span> <span class="s">"https://sandboxapicdc.cisco.com/api/"</span> <span class="c1">## Calling the authentication function above to get the auth cookie </span><span class="n">cookies</span> <span class="o">=</span> <span class="n">get_token</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">base_url</span><span class="p">)</span> <span class="n">pprint</span><span class="p">(</span><span class="n">pull_pod1</span><span class="p">(</span><span class="n">cookies</span><span class="p">,</span> <span class="n">base_url</span><span class="p">))</span> </code></pre></div></div> <p>This gives us all the devices in POD1. I’ll just be putting in one device for the sake of space</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="s">"imdata"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"fabricNode"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"attributes"</span><span class="p">:</span> <span class="p">{</span> <span class="s">"adSt"</span><span class="p">:</span> <span class="s">"on"</span><span class="p">,</span> <span class="s">"address"</span><span class="p">:</span> <span class="s">"10.0.112.64"</span><span class="p">,</span> <span class="s">"annotation"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"apicType"</span><span class="p">:</span> <span class="s">"apic"</span><span class="p">,</span> <span class="s">"childAction"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"delayedHeartbeat"</span><span class="p">:</span> <span class="s">"no"</span><span class="p">,</span> <span class="s">"dn"</span><span class="p">:</span> <span class="s">"topology/pod-1/node-101"</span><span class="p">,</span> <span class="s">"extMngdBy"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"fabricSt"</span><span class="p">:</span> <span class="s">"active"</span><span class="p">,</span> <span class="s">"id"</span><span class="p">:</span> <span class="s">"101"</span><span class="p">,</span> <span class="s">"lastStateModTs"</span><span class="p">:</span> <span class="s">"2019-12-12T14:22:14.238+00:00"</span><span class="p">,</span> <span class="s">"lcOwn"</span><span class="p">:</span> <span class="s">"local"</span><span class="p">,</span> <span class="s">"modTs"</span><span class="p">:</span> <span class="s">"2019-12-12T14:23:02.315+00:00"</span><span class="p">,</span> <span class="s">"model"</span><span class="p">:</span> <span class="s">"N9K-C9396PX"</span><span class="p">,</span> <span class="s">"monPolDn"</span><span class="p">:</span> <span class="s">"uni/fabric/monfab-default"</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"leaf-1"</span><span class="p">,</span> <span class="s">"nameAlias"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"nodeType"</span><span class="p">:</span> <span class="s">"unspecified"</span><span class="p">,</span> <span class="s">"role"</span><span class="p">:</span> <span class="s">"leaf"</span><span class="p">,</span> <span class="s">"serial"</span><span class="p">:</span> <span class="s">"TEP-1-101"</span><span class="p">,</span> <span class="s">"status"</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="s">"uid"</span><span class="p">:</span> <span class="s">"0"</span><span class="p">,</span> <span class="s">"vendor"</span><span class="p">:</span> <span class="s">"Cisco Systems, Inc"</span><span class="p">,</span> <span class="s">"version"</span><span class="p">:</span> <span class="s">""</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> </code></pre></div></div> <blockquote> <p>Note: Output omitted and modified for the sake of brevity and clarity.</p> </blockquote> <p>You can observe several key pieces of information about this switch. The DN info, ID, model number, serial number, etc. With ACI, I often had to pull one piece of data so that I could understand how to pull another piece of data. Finding the DN was almost always an important step in my programs.</p> <p>There is an easier, albeit less stable way (based on my observations) to find the URLs we want. Right-click on an interesting object and select “Open in Object Store Browser.” That will direct to a login screen. After login, the UI brings up a page where it should display the query and give the output.</p> <center><img src="../../../static/images/basic_api_use_with_python/open_in_object_browser.jpg" /></center> <p>To see the URL it used we click on “Show URL and response of last query.”</p> <center><img src="../../../static/images/basic_api_use_with_python/See_url_from_browser.jpg" /></center> <p>You can navigate the API structure and find what you are looking for. It may take some practice to develop a full understanding of this method.</p> <p>I hope this has been helpful. I know it can be overwhelming, but the benefits of working with an API vs clicking though a web-page are wonderful. For example, before I left my last job I wrote a Python program that could be run on demand. It pulled what VLANS went to what VM Servers and looked for errors on interfaces. We got a call that there were lots of outages going on. The other network engineers were manually logging into everything looking for issues, but I just ran my program and was able to find a missing VLAN on a trunk to a VM server. The VM failed because it could no longer communicate out-side of the VM server. It took approximately five minutes to find and fix the issue. With dozens of VM servers and very little info on what the actual issue was, I believe that outage would have been much longer if people were looking for the issue by hand.</p> <p>-Daniel</p>Daniel HimesI come from a network engineering background, so when I started my network automation journey, it felt like I was banging my head against new technology with very little help. When I first started using APIs, our network team had just started with this brand new ACI “thing,” and I couldn’t find a well written “What is an API” blog with code I would call “appropriate.” In this blog, I will attempt to provide that for those of you that are just getting started with APIs. If you have been using APIs for a while, this may not be as useful.Using NTC Templates in Ansible2020-02-04T00:00:00+00:002020-02-04T00:00:00+00:00https://blog.networktocode.com/post/using_ntc_templates<p>Network to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created <a href="https://github.com/google/textfsm">TextFSM</a> that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a <code class="highlighter-rouge">show lldp neighbor</code> command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.</p> <p>You can find more about the <a href="https://github.com/networktocode/ntc-templates">ntc-templates on our Github page</a>. These are templates that have been created by both Network to Code and by the community. It is an open source project that anyone is able to contribute a template to if there is not a template already created. Take a look at the <a href="https://github.com/networktocode/ntc-templates">readme</a> for how you can help contribute to NTC Templates!</p> <h2 id="other-methods">Other Methods</h2> <p>Ansible is not the only method for using TextFSM to get structured data returned from network device output. Several Python libraries such as Netmiko and Nornir are able to use TextFSM to return structured data by setting a flag to use TextFSM as well.</p> <h2 id="using-ntc-templates-with-ansible">Using NTC-Templates with Ansible</h2> <p>There are two primary methods for sending data through a TextFSM parser in Ansible. You will get to see examples for both of these. The first method available is to use the TextFSM Filter. The second is in conjunction with the Ansible Galaxy role Network Engine.</p> <h3 id="getting-started">Getting Started</h3> <p>The first step towards leveraging these templates is getting the templates on to a compute device that is running Ansible. For this demo, we’ll be using a local machine and Git to clone the repository to a local directory.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:networktocode/ntc-templates.git </code></pre></div></div> <p>Next you will need to install TextFSM to the same Python interpreter that your Ansible installation is installed in. This demo will leverage Python3. For the installation you will also use the <code class="highlighter-rouge">--user</code> flag to install as the local user and <code class="highlighter-rouge">--upgrade</code> to ensure that the latest version of TextFSM is installed.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip3 <span class="nb">install </span>textfsm <span class="nt">--user</span> <span class="nt">--upgrade</span> </code></pre></div></div> <p>For the second method of using Ansible Galaxy’s Network Engine you will also need to install the role as follows:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-galaxy <span class="nb">install </span>ansible-network.network-engine </code></pre></div></div> <p>Now that the environment is setup it’s time to take a look at the Playbook and the execution.</p> <h3 id="playbook-setup-and-execution">Playbook Setup and Execution</h3> <p>This lab is setup as a 3 router traingle. RTR-1 will connect to one interface on RTR-2 and RTR-3. RTR-2 will have one connection to RTR-1 and RTR-3. RTR-3 will have the corresponding connections to RTR-1 and RTR-2.</p> <p><img src="../../../static/images/blog_posts/ansible-using-ntc-templates/using_templates_3router.png" alt="3_router_triangle" /></p> <h4 id="textfsm-cli-parser---ansible-built-in">TextFSM CLI Parser - Ansible Built In</h4> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">DEMO</span><span class="nv"> </span><span class="s">OF</span><span class="nv"> </span><span class="s">TEXTFSM"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">routers</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">COMMAND</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">show lldp neighbors</span> <span class="na">register</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">REGISTER</span><span class="nv"> </span><span class="s">OUTPUT</span><span class="nv"> </span><span class="s">TO</span><span class="nv"> </span><span class="s">DEVICE_NEIGHBORS</span><span class="nv"> </span><span class="s">VARIABLE"</span> <span class="na">set_fact</span><span class="pi">:</span> <span class="na">device_neighbors</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lldp_output.stdout[0]</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">parse_cli_textfsm('~/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm')</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">3:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_neighbors</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">4:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">NEIGHBORS"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}:</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item['NEIGHBOR']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">device_neighbors</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>Walking through this first play, in <em>Task 1</em> you are connecting to the device and running the command <code class="highlighter-rouge">show lldp neighbors</code>. This is saved to a variable named <code class="highlighter-rouge">lldp_output</code> that will be used later.</p> <p>In <em>Task 2</em> the output is being sent through the <strong>parse_cli_textfsm</strong> filter. The filter is being provided the library file to be used in the parsing. This is explicitely called out. The output from going through the TextFSM parser is then getting assigned to the variable <code class="highlighter-rouge">device_neighbors</code>, with each host in the execution having their own local instance of this variable.</p> <p>In <em>Task 3</em> you are getting to see the output of the variable. This shows the structured data. With this particular template you get the keys of <strong>CAPABILITIES</strong>, <strong>LOCAL_INTERFACES</strong>, <strong>NEIGHBOR</strong>, and <strong>NEIGHBOR_INTERFACE</strong> back. These are all defined within the TextFSM template.</p> <p>In <em>Task 4</em> you get to see a practical output to a screen if you want to audit and understand the neighbor relationships seen by LLDP on the network devices.</p> <p>In this example you need to set a fact to be able to access the data later or continue to send the output through the <code class="highlighter-rouge">parse_cli_textfsm</code> filter every time you want to get at the structured data, such as a line number 15 in TASK 2.</p> <h5 id="textfsm-cli-parser-output">TextFSM CLI Parser Output</h5> <p>Here is the output that corresponds with the play above.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-playbook demo_ansible_textfsm.yml PLAY <span class="o">[</span>PLAY 1: DEMO OF TEXTFSM WITH CLI PARSER] <span class="k">*************************************************************************</span> TASK <span class="o">[</span>TASK 1: GET COMMAND OUTPUT] <span class="k">**************************************************************************************</span> ok: <span class="o">[</span>rtr-2] ok: <span class="o">[</span>rtr-3] ok: <span class="o">[</span>rtr-1] TASK <span class="o">[</span>TASK 2: REGISTER OUTPUT TO DEVICE_NEIGHBORS VARIABLE] <span class="k">************************************************************</span> ok: <span class="o">[</span>rtr-1] ok: <span class="o">[</span>rtr-2] ok: <span class="o">[</span>rtr-3] TASK <span class="o">[</span>TASK 3: PRINT OUTPUT] <span class="k">********************************************************************************************</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/0"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-3"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/0"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-1"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/2"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-3"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-2"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/1"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="o">[</span> <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/1"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-1"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/2"</span> <span class="o">}</span>, <span class="o">{</span> <span class="s2">"CAPABILITIES"</span>: <span class="s2">"R"</span>, <span class="s2">"LOCAL_INTERFACE"</span>: <span class="s2">"Gi0/0"</span>, <span class="s2">"NEIGHBOR"</span>: <span class="s2">"rtr-2"</span>, <span class="s2">"NEIGHBOR_INTERFACE"</span>: <span class="s2">"Gi0/0"</span> <span class="o">}</span> <span class="o">]</span> <span class="o">}</span> TASK <span class="o">[</span>TASK 4: PRINT NEIGHBORS FROM CLI PARSER] <span class="k">*************************************************************************</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/2<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/2: rtr-3"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-1] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-2"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/0<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/0: rtr-3"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-2] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-1"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/1<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/1: rtr-1"</span> <span class="o">}</span> ok: <span class="o">[</span>rtr-3] <span class="o">=&gt;</span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>Gi0/0<span class="o">)</span> <span class="o">=&gt;</span> <span class="o">{</span> <span class="s2">"msg"</span>: <span class="s2">"Gi0/0: rtr-2"</span> <span class="o">}</span> PLAY RECAP <span class="k">*************************************************************************************************************</span> rtr-1 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 rtr-2 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 rtr-3 : <span class="nv">ok</span><span class="o">=</span>4 <span class="nv">changed</span><span class="o">=</span>0 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0 <span class="nv">skipped</span><span class="o">=</span>0 <span class="nv">rescued</span><span class="o">=</span>0 <span class="nv">ignored</span><span class="o">=</span>0 </code></pre></div></div> <h4 id="ansible-role---network-engine">Ansible Role - Network Engine</h4> <p>The Ansible Role - Network Engine is a role that extracts information about network devices into Ansible Facts. You can use either TextFSM syntax or YAML (<code class="highlighter-rouge">command_parser</code> option) to extract information about a network device from the command output. You can read more about using the role at the <a href="https://github.com/ansible-network/network-engine/blob/devel/docs/user_guide/README.md">network-engine Github page</a>.</p> <p>Here you will see a second method to get the same output with Network Engine instead of the CLI Parser. The biggest difference is that in this method, the Network Engine role will register the returned data to the ansible_facts instead of requiring you to set it to your own variable.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">PLAY</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">DEMO</span><span class="nv"> </span><span class="s">OF</span><span class="nv"> </span><span class="s">TEXTFSM</span><span class="nv"> </span><span class="s">WITH</span><span class="nv"> </span><span class="s">NETWORK</span><span class="nv"> </span><span class="s">ENGINE"</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">routers</span> <span class="na">connection</span><span class="pi">:</span> <span class="s">network_cli</span> <span class="na">gather_facts</span><span class="pi">:</span> <span class="s">no</span> <span class="na">roles</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">ansible-network.network-engine</span> <span class="na">tasks</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">1:</span><span class="nv"> </span><span class="s">GET</span><span class="nv"> </span><span class="s">COMMAND</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">ios_command</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">show lldp neighbors</span> <span class="na">register</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">2:</span><span class="nv"> </span><span class="s">RUN</span><span class="nv"> </span><span class="s">THROUGH</span><span class="nv"> </span><span class="s">THE</span><span class="nv"> </span><span class="s">PARSER"</span> <span class="na">textfsm_parser</span><span class="pi">:</span> <span class="na">file</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/Users/ntcblog/ntc-templates/templates/cisco_ios_show_lldp_neighbors.textfsm"</span> <span class="na">content</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">lldp_output.stdout[0]</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">name</span><span class="pi">:</span> <span class="s">lldp_output</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">3:</span><span class="nv"> </span><span class="s">SHOW</span><span class="nv"> </span><span class="s">ANSIBLE</span><span class="nv"> </span><span class="s">FACTS</span><span class="nv"> </span><span class="s">OUTPUT"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts</span><span class="nv"> </span><span class="s">}}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">TASK</span><span class="nv"> </span><span class="s">4:</span><span class="nv"> </span><span class="s">PRINT</span><span class="nv"> </span><span class="s">NEIGHBORS</span><span class="nv"> </span><span class="s">FROM</span><span class="nv"> </span><span class="s">ANSIBLE</span><span class="nv"> </span><span class="s">NETWORK</span><span class="nv"> </span><span class="s">ENGINE"</span> <span class="na">debug</span><span class="pi">:</span> <span class="na">msg</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}:</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item['NEIGHBOR']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">ansible_facts['lldp_output']</span><span class="nv"> </span><span class="s">}}"</span> <span class="na">loop_control</span><span class="pi">:</span> <span class="na">label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{{</span><span class="nv"> </span><span class="s">item['LOCAL_INTERFACE']</span><span class="nv"> </span><span class="s">}}"</span> </code></pre></div></div> <p>There is a new key that we needed to add to import the role that was installed with the <code class="highlighter-rouge">ansible-galaxy</code> command earlier. This is the <code class="highlighter-rouge">roles:</code> key that you see under <code class="highlighter-rouge">gather_facts</code> and contains a list of roles to import into the play. Here you see the import of the <code class="highlighter-rouge">ansible-network.network-engine</code> role.</p> <p>In <em>Task 1</em> you once again have the same command to gather the LLDP neighbors from the device.</p> <p>In <em>Task 2</em> instead of registering to a fact and sending through a filter, you now use the module <code class="highlighter-rouge">textfsm_parser</code> that takes <em>file</em> as a parameter that is the file of the TextFSM template. This is the same template referenced in the filter on the first play. You also pass <em>content</em> of what output you want to send through the parser. The last parameter is the <em>name</em> of the fact that you are registering to ansible_facts.</p> <p>In <em>Task 3</em>, you once again get to see the output for <code class="highlighter-rouge">ansible_facts</code>. This will show that there are more details available about the device.</p> <p>In <em>Task 4</em> you get to see the same output as <em>Play 1</em> where you get the neighbors printed out.</p> <h5 id="network-engine-output">Network Engine Output</h5> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">ansible-playbook demo_ansible_textfsm.yml</span> <span class="s">PLAY [PLAY 2</span><span class="pi">:</span> <span class="s">DEMO OF TEXTFSM WITH NETWORK ENGINE] *********************************************************************</span> <span class="s">TASK [TASK 1</span><span class="pi">:</span> <span class="s">GET COMMAND OUTPUT] **************************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">TASK [TASK 2</span><span class="pi">:</span> <span class="s">RUN THROUGH THE PARSER] **********************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">TASK [TASK 3</span><span class="pi">:</span> <span class="s">SHOW ANSIBLE FACTS OUTPUT] *******************************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-3"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-3"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">discovered_interpreter_python"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/usr/bin/python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">lldp_output"</span><span class="pi">:</span> <span class="pi">[</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-1"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2"</span> <span class="pi">},</span> <span class="pi">{</span> <span class="s2">"</span><span class="s">CAPABILITIES"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">R"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">LOCAL_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">rtr-2"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">NEIGHBOR_INTERFACE"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0"</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span> <span class="err">}</span> <span class="s">TASK [TASK 4</span><span class="pi">:</span> <span class="s">PRINT NEIGHBORS FROM ANSIBLE NETWORK ENGINE] *************************************************************</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/2) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/2:</span><span class="nv"> </span><span class="s">rtr-3"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-1</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-2"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/0) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0:</span><span class="nv"> </span><span class="s">rtr-3"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-1"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-2</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/1) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/1:</span><span class="nv"> </span><span class="s">rtr-1"</span> <span class="err">}</span> <span class="na">ok</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">rtr-3</span><span class="pi">]</span> <span class="s">=&gt; (item=Gi0/0) =&gt; {</span> <span class="s">"msg"</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Gi0/0:</span><span class="nv"> </span><span class="s">rtr-2"</span> <span class="err">}</span> <span class="s">PLAY RECAP *************************************************************************************************************</span> <span class="na">rtr-1 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> <span class="na">rtr-2 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> <span class="na">rtr-3 </span><span class="pi">:</span> <span class="s">ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0</span> </code></pre></div></div> <h2 id="summary">Summary</h2> <p>This shows that there are multiple ways of pairing TextFSM templates with Ansible. Getting structured data out of unstructured output is extremely valuable when it comes to automating a network environment. There are over 300 different templates currently in the NTC-Templates repository to help get structured data out of your unstructured data.</p> <p>-Josh</p>Josh VanDeraaNetwork to Code has a repository created for maintaining templates that help to convert unstructured output from network devices into structured data. Google created TextFSM that provides a custom DSL to take command line output and return as structured data. Examples range from getting an output from a show lldp neighbor command and returning the list of neighbors to getting a list of addresses in an ARP table. This post is going to dive deeper into how to use these to get the data with Ansible.Versionable Database2020-01-28T00:00:00+00:002020-01-28T00:00:00+00:00https://blog.networktocode.com/post/Versionable-Database<p>One of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.</p> <h2 id="problem-space">Problem Space</h2> <p>When managing and storing your Source of Truth data, there are two primary mechanisms, each with it’s own pros and cons.</p> <p>Source Control (Git)</p> <ul> <li>Provides the ability to version data in such a way the exact state of the data at any point in the past is known.</li> <li>Provides the ability to know who is the owner of the data.</li> <li>Provides the ability to populate data in a staging area, without modifying the production data.</li> <li>Tooling integration with things such as CI Systems.</li> <li>Decentralized storage of the data.</li> </ul> <p>Database</p> <ul> <li>Provides <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>-based transactions of the data.</li> <li>Has a native querying language to obtain, filter, and all around work with the data.</li> <li>Provides schema enforcement of the data.</li> <li>The ability to scale to large datasets.</li> </ul> <p>There is a clear dichotomy between these two choices. On one hand there is great integrations with all standard DevOps tooling, on the other hand there is an enterprise-grade manager of the data.</p> <h2 id="history">History</h2> <p>For several years, a few of us have been searching for solutions that combine these two concepts. If one could build a database with Git constructs, this would allow versionable management of your data, with the ability to query, effectively store, and manage at scale. Through various searches it would seem that this has not been solved in any meaningful way. The closest I have come across was a project called <a href="https://github.com/attic-labs/noms">NOMS</a>, but it has limitations.</p> <p>Recently a startup company called <a href="https://www.liquidata.co/">liquidata</a> was founded and is creating a solution–a library called <a href="https://github.com/liquidata-inc/dolt">Dolt</a>. Dolt is is written on top of the NOMS project, in Go. The intention is to support all Git semantics (such as branch, add, commit, etc..), as well as all MySQL semantics (such as insert, create, update, etc…).</p> <p>By supporting all MySQL semantics, applications that use a MySQL server should be able to simply use a Dolt-SQL server with no disruption. Whether you are using an ODBC driver or raw SQL, in theory it should still work. You would naturally have to build in the capability for your application to take advantage of the Git capabilities or use traditional “Git-like” workflows.</p> <p>By supporting all Git semantics, data should be able to be managed in a decentralized fashion using Git workflows to branch, add, diff, and merge the data. In a similar vein to GitHub, they have developed DoltHub, to provide that level of tooling expected in a standard Git user interface, such as forking, API’s, webhooks, and CI integrations.</p> <h2 id="primer">Primer</h2> <p>In this brief introduction to the technology, we will create a Dolt data repository, called “simple-inventory” and run it on a dolt-sql server. If you care to follow along the only requirements are to have Docker and internet access.</p> <h2 id="setup">Setup</h2> <p>Start the Docker container.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="nt">--name</span> dolt golang:1.12.14 </code></pre></div></div> <blockquote> <p>Install Dolt by running <code class="highlighter-rouge">curl -L https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash</code></p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go# curl <span class="nt">-L</span> https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/install.sh | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 601 0 601 0 0 2647 0 <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- 2647 100 3038 100 3038 0 0 8392 0 <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- <span class="nt">--</span>:--:-- 8392 Downloading: https://github.com/liquidata-inc/dolt/releases/download/v0.12.0/dolt-linux-amd64.tar.gz Installing dolt, git-dolt and git-dolt-smudge to /usr/local/bin. root@2a29f8a34dd3:/go# </code></pre></div></div> <p>You can create a Dolt data repository by creating a folder, and then initializing the repository. Just like Git, in Dolt, you need to add yourself to the Dolt config. If you are familiar with Git, these commands will be familiar, only changing the command from <code class="highlighter-rouge">git</code> to <code class="highlighter-rouge">dolt</code>, but keeping all of the other options the same.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go# <span class="nb">mkdir </span>simple-inventory root@2a29f8a34dd3:/go# <span class="nb">cd </span>simple-inventory root@2a29f8a34dd3:/go/simple-inventory# dolt config <span class="nt">--global</span> <span class="nt">--add</span> user.email <span class="s2">"ken@celenza.org"</span> Config successfully updated. root@2a29f8a34dd3:/go/simple-inventory# dolt config <span class="nt">--global</span> <span class="nt">--add</span> user.name <span class="s2">"itdependsnetworks"</span> Config successfully updated. root@2a29f8a34dd3:/go/simple-inventory# dolt init Successfully initialized dolt data repository. root@2a29f8a34dd3:/go/simple-inventory# dolt status On branch master nothing to commit, working tree clean root@2a29f8a34dd3:/go/simple-inventory# dolt log commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <h2 id="creating-tables">Creating Tables</h2> <p>Ideally, if you are familiar with MySQL the only difference should be using Dolt engine instead of a MySQL server, everything should be the same. To enter the Dolt SQL server, simply use the command <code class="highlighter-rouge">dolt sql</code>.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <p>Create tables as you normally would, in this example, we will create two tables. The table <code class="highlighter-rouge">device_inventory</code> will have the inventory of the devices, with columns for hostname and IP address. The table <code class="highlighter-rouge">vlan</code> will have the hostname (with foreign key relationaship to the <code class="highlighter-rouge">device_inventory</code> table) column, VLAN , and name of the VLAN.</p> <blockquote> <p>Note: Though the foreign key relationaship syntax is defined, it is not a currently supported Dolt feature.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">device_inventory</span> <span class="p">(</span> <span class="o">-&gt;</span> <span class="n">hostname</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="n">ip_address</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="k">primary</span> <span class="k">key</span> <span class="p">(</span><span class="nv">`hostname`</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">);</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">vlan</span> <span class="p">(</span> <span class="o">-&gt;</span> <span class="n">hostname</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">),</span> <span class="o">-&gt;</span> <span class="n">vlan</span> <span class="nb">int</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="n">name</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="o">-&gt;</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">),</span> <span class="o">-&gt;</span> <span class="k">FOREIGN</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">hostname</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">device_inventory</span><span class="p">(</span><span class="n">hostname</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">);</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> </code></pre></div></div> <p>Here is where it starts to get interesting–we can combine these two different concepts and see how they interact. So far, the data has not been committed into master, only staged into our local environment. We can view this by issuing standard Git-like commands when we return back to the command line from the dolt-sql server.</p> <blockquote> <p>Run the <code class="highlighter-rouge">dolt status</code>, <code class="highlighter-rouge">dolt diff</code>, and <code class="highlighter-rouge">dolt diff -q</code> commands.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt status On branch master Untracked files: <span class="o">(</span>use <span class="s2">"dolt add &lt;table&gt;"</span> to include <span class="k">in </span>what will be committed<span class="o">)</span> new table: device_inventory new table: vlan root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/device_inventory b/device_inventory added table diff <span class="nt">--dolt</span> a/vlan b/vlan added table root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> CREATE TABLE <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span> <span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:0'</span>, <span class="sb">`</span>ip_address<span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:1'</span>, PRIMARY KEY <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">)</span> <span class="o">)</span><span class="p">;</span> CREATE TABLE <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span> <span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:0'</span>, <span class="sb">`</span>vlan<span class="sb">`</span> BIGINT NOT NULL COMMENT <span class="s1">'tag:1'</span>, <span class="sb">`</span>name<span class="sb">`</span> LONGTEXT NOT NULL COMMENT <span class="s1">'tag:2'</span>, PRIMARY KEY <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span><span class="o">)</span> <span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>The <code class="highlighter-rouge">dolt status</code> command shows that the schema has not “taken effect” by being merged into master. The <code class="highlighter-rouge">dolt diff</code> with optional <code class="highlighter-rouge">-q</code> flag, shows that a new table has been created. Note the additional <code class="highlighter-rouge">tag</code> parameters, this is logic to track column names, which may change over time and cause name conflicts. The new schema can be committed into master.</p> <blockquote> <p>Issue the commands <code class="highlighter-rouge">dolt add -a</code>, <code class="highlighter-rouge">dolt commit -m 'initial schema'</code>, and <code class="highlighter-rouge">dolt log</code>.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'initial schema'</span> commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema root@2a29f8a34dd3:/go/simple-inventory# dolt log commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>Awesome! The first piece of information of the database was committed.</p> <h2 id="adding-data">Adding data</h2> <p>Defining schema is not all the valuable without the data. Again, using standard SQL constructs we can commit the data. Let’s use proper branching strategies this time.</p> <blockquote> <p>Create a branch and Enter dolt-sql</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout <span class="nt">-b</span> intial_data Switched to branch <span class="s1">'intial_data'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>Add data</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">device_inventory</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">ip_address</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="nv">"10.1.1.1"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">device_inventory</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">ip_address</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="nv">"10.1.1.2"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">"user"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="nv">"printer"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw01"</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="nv">"wap"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">"user"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="nv">"printer"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">vlan</span> <span class="p">(</span><span class="n">hostname</span><span class="p">,</span> <span class="n">vlan</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="nv">"nyc-sw02"</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="nv">"wap"</span><span class="p">);</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> <span class="n">Bye</span> </code></pre></div></div> <p>Now that we have staged the data, we can view the date in two different ways. The first is via raw sql entries, and the other is more akin to a unix diff.</p> <blockquote> <p>View the diff. In the console, diffs are actually color coded green and red for add and remove respectively.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/device_inventory b/device_inventory <span class="nt">---</span> a/device_inventory @ tktob1spsoos6isfdj4o9benp9c57iic +++ b/device_inventory @ b37haqr0n9j1e5bckj35sqt4kntojlp9 +-----+----------+------------+ | | <span class="nb">hostname</span> | ip_address | +-----+----------+------------+ | + | nyc-sw01 | 10.1.1.1 | | + | nyc-sw02 | 10.1.1.2 | +-----+----------+------------+ diff <span class="nt">--dolt</span> a/vlan b/vlan <span class="nt">---</span> a/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn +++ b/vlan @ e2kovelv2sfjuo584o0v6hp76207k720 +-----+----------+------+---------+ | | <span class="nb">hostname</span> | vlan | name | +-----+----------+------+---------+ | + | nyc-sw01 | 10 | user | | + | nyc-sw01 | 20 | printer | | + | nyc-sw01 | 30 | wap | | + | nyc-sw02 | 10 | user | | + | nyc-sw02 | 20 | printer | | + | nyc-sw02 | 30 | wap | +-----+----------+------+---------+ root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> INSERT INTO <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>ip_address<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,<span class="s2">"10.1.1.1"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>device_inventory<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>ip_address<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,<span class="s2">"10.1.1.2"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,10,<span class="s2">"user"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,20,<span class="s2">"printer"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw01"</span>,30,<span class="s2">"wap"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,10,<span class="s2">"user"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,20,<span class="s2">"printer"</span><span class="o">)</span><span class="p">;</span> INSERT INTO <span class="sb">`</span>vlan<span class="sb">`</span> <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span>,<span class="sb">`</span>vlan<span class="sb">`</span>,<span class="sb">`</span>name<span class="sb">`</span><span class="o">)</span> VALUES <span class="o">(</span><span class="s2">"nyc-sw02"</span>,30,<span class="s2">"wap"</span><span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>Personally, I’m pretty impressed with what is happening here, but there is still more to do to complete this workflow. The data needs to be committed, and merged from the “feature” branch into master branch.</p> <blockquote> <p>Commit to the feature branch.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'initial data'</span> commit gbt6r21mtbd0cvs5iahog912essjhn0n Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:50 +0000 2020 initial data root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <blockquote> <p>Merge into the master branch.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master Switched to branch <span class="s1">'master'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt branch intial_data <span class="k">*</span> master root@2a29f8a34dd3:/go/simple-inventory# dolt merge intial_data Updating 4n29eaqb1v9pmnpnda2r43fc91p1vp9l..gbt6r21mtbd0cvs5iahog912essjhn0n Fast-forward root@2a29f8a34dd3:/go/simple-inventory# dolt log commit gbt6r21mtbd0cvs5iahog912essjhn0n Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:50 +0000 2020 initial data commit 4n29eaqb1v9pmnpnda2r43fc91p1vp9l Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:41 +0000 2020 initial schema commit hc8gq0434ckkjeo36rccn55mo14gk87q Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:22:33 +0000 2020 Initialize data repository root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>So far this is showing the “Create” part of standard CRUD operations is working, meaning we can add data to the repository.</p> <h2 id="update-data">Update Data</h2> <p>Naturally, as soon as data is entered, you will want to modify it. The process is the same to modify: branch, sql statements, add, commit, and merge.</p> <blockquote> <p>Checkout a new branch, and enter Dolt SQL.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt checkout <span class="nt">-b</span> change_vlan Switched to branch <span class="s1">'change_vlan'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>Update the data.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">update</span> <span class="n">vlan</span> <span class="k">set</span> <span class="n">name</span> <span class="o">=</span> <span class="nv">"prnt"</span> <span class="k">where</span> <span class="n">name</span> <span class="o">=</span> <span class="nv">"printer"</span><span class="p">;</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="o">|</span> <span class="n">matched</span> <span class="o">|</span> <span class="n">updated</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="o">+</span><span class="c1">---------+---------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> <span class="n">exit</span> <span class="n">Bye</span> </code></pre></div></div> <blockquote> <p>View the diff.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt diff diff <span class="nt">--dolt</span> a/vlan b/vlan <span class="nt">---</span> a/vlan @ bjg5ou111lmu03ciu6aisphf7m6jbk1r +++ b/vlan @ 7f8lqlv2k9cpdth68kob9hl8atuejrpn +-----+----------+------+---------+ | | <span class="nb">hostname</span> | vlan | name | +-----+----------+------+---------+ | &lt; | nyc-sw01 | 20 | printer | | <span class="o">&gt;</span> | nyc-sw01 | 20 | prnt | | &lt; | nyc-sw02 | 20 | printer | | <span class="o">&gt;</span> | nyc-sw02 | 20 | prnt | +-----+----------+------+---------+ root@2a29f8a34dd3:/go/simple-inventory# dolt diff <span class="nt">-q</span> UPDATE <span class="sb">`</span>vlan<span class="sb">`</span> SET <span class="sb">`</span>name<span class="sb">`</span><span class="o">=</span><span class="s2">"prnt"</span> WHERE <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">=</span><span class="s2">"nyc-sw01"</span> AND <span class="sb">`</span>vlan<span class="sb">`</span><span class="o">=</span>20<span class="o">)</span><span class="p">;</span> UPDATE <span class="sb">`</span>vlan<span class="sb">`</span> SET <span class="sb">`</span>name<span class="sb">`</span><span class="o">=</span><span class="s2">"prnt"</span> WHERE <span class="o">(</span><span class="sb">`</span><span class="nb">hostname</span><span class="sb">`</span><span class="o">=</span><span class="s2">"nyc-sw02"</span> AND <span class="sb">`</span>vlan<span class="sb">`</span><span class="o">=</span>20<span class="o">)</span><span class="p">;</span> root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <p>A keen eye will note the captured update statements shown in the diff are not the same as the update statement inputted in the SQL server. This conversion makes sense, since the data could be merged on a different set of data that had more or less data in it. It also illustrates the complexity of building this technology.</p> <blockquote> <p>Commit to a feature branch and merge to master.</p> </blockquote> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt add <span class="nt">-a</span> root@2a29f8a34dd3:/go/simple-inventory# dolt commit <span class="nt">-m</span> <span class="s1">'update data'</span> commit jqe3vb84fhfunn7uapksdt5thb4rr7r7 Author: itdependsnetworks &lt;ken@celenza.org&gt; Date: Tue Jan 28 04:23:03 +0000 2020 update data root@2a29f8a34dd3:/go/simple-inventory# dolt checkout master Switched to branch <span class="s1">'master'</span> root@2a29f8a34dd3:/go/simple-inventory# dolt merge change_vlan Updating gbt6r21mtbd0cvs5iahog912essjhn0n..jqe3vb84fhfunn7uapksdt5thb4rr7r7 Fast-forward root@2a29f8a34dd3:/go/simple-inventory# dolt sql <span class="c"># Welcome to the DoltSQL shell.</span> <span class="c"># Statements must be terminated with ';'.</span> <span class="c"># "exit" or "quit" (or Ctrl-D) to exit.</span> doltsql&gt; </code></pre></div></div> <blockquote> <p>View the data from dolt sql.</p> </blockquote> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doltsql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">vlan</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="o">|</span> <span class="n">hostname</span> <span class="o">|</span> <span class="n">vlan</span> <span class="o">|</span> <span class="n">name</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="k">user</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">20</span> <span class="o">|</span> <span class="n">prnt</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw01</span> <span class="o">|</span> <span class="mi">30</span> <span class="o">|</span> <span class="n">wap</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="k">user</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">20</span> <span class="o">|</span> <span class="n">prnt</span> <span class="o">|</span> <span class="o">|</span> <span class="n">nyc</span><span class="o">-</span><span class="n">sw02</span> <span class="o">|</span> <span class="mi">30</span> <span class="o">|</span> <span class="n">wap</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+------+------+</span> <span class="n">doltsql</span><span class="o">&gt;</span> </code></pre></div></div> <p>You can also view who is the owner of the data by tracking to the commit, which includes the auther, time, and commit hash metadata as well, using the <code class="highlighter-rouge">dolt blame &lt;table&gt;</code> command.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@2a29f8a34dd3:/go/simple-inventory# dolt blame vlan +----------+------+--------------+-------------------+------------------------------+-------------------+ | HOSTNAME | VLAN | COMMIT MSG | AUTHOR | TIME | COMMIT | +----------+------+--------------+-------------------+------------------------------+-------------------+ | nyc-sw01 | 30 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw02 | 10 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw02 | 20 | update data | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua | | nyc-sw02 | 30 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw01 | 10 | initial data | itdependsnetworks | Tue Jan 28 04:22:50 UTC 2020 | gbt6r21mtbd0cvs5i | | nyc-sw01 | 20 | update data | itdependsnetworks | Tue Jan 28 04:23:03 UTC 2020 | jqe3vb84fhfunn7ua | +----------+------+--------------+-------------------+------------------------------+-------------------+ root@2a29f8a34dd3:/go/simple-inventory# </code></pre></div></div> <h2 id="summary">Summary</h2> <p>This is just a primer, and there is still a lot of work to be done to truly have feature parity with both MySQL and Git, but this is a great step in the right direction. I plan to continue to monitor and follow up with examples, use cases, and library updates over time. Specifically, next time, I want to extend the workflow to include DoltHub to review those capabilities as well.</p> <p>-Ken</p>Ken CelenzaOne of the most important components of a network automation strategy is how to manage your data and/or Source of Truth (SoT). In networking, the data elements are often massive. As an example, most Fortune 500 type companies will have to manage a few hundred thousand to potentially millions of switchports, each with several data points (VLAN, MTU, IP, etc) per interface. The scale and criticality of this data present several challenges. The intention of this blog is describe the need for a versionable database in network automation and serve as an primer to a versionable database.