Skip to content

Mission 4: Use pyATS to verify your automated deployment

What is pyATS and Genie?

pyATS (Python Automated Test System) is a testing framework developed by Cisco for automating network testing and validation. It helps ensure that your network devices and configurations are functioning as expected by running automated test cases. With pyATS, you can perform tasks such as verifying deployments, checking configurations, and troubleshooting issues.

Genie (Generate Easy Network Information Engine) is a library that works alongside pyATS to simplify network testing and automation. It provides:

  1. Parsers: Pre-built tools to extract and structure information from device outputs (e.g., CLI commands like show ip interface brief). These parsers turn raw text into structured Python objects, making it easier to analyze and manipulate the data.
  2. Triggers: Pre-built functions that automate specific tasks or operations on devices.
  3. Testbeds: A way to define and manage your network topology. Testbeds describe your network devices and their properties, such as:
    • Device roles (e.g., router, switch).
    • Connection details (e.g., SSH, Telnet, or Console).
    • Credentials (e.g., username and password).

By defining your Testbed , you can easily manage device connections and run various tests across your network.

What do we need to use pyATS?

To use pyATS, there are a few key components you need:

  1. Testbed

    A testbed is a YAML file that describes your network environment. It contains information about the devices under test, such as their IP addresses, credentials, and connection methods.

    Example Pod-1:

    testbed:
  credentials:
    default:
      username: netadmin
      password: C1sco12345

devices:
  WLSN-sw1:
    os: iosxe
    connections:
      vty:
        protocol: ssh
        ip: 10.99.1.11

  2. Test script

    This is a Python script that defines what pyATS should do. It typically includes the following sections:

    • CommonSetup: Prepares the environment before running the actual tests (e.g., connecting to devices).
    • Testcases: Contains the actual tests or validations to be performed.
    • CommonCleanup: Cleans up resources or restores the environment after the tests are done (e.g., disconnecting from devices).

    Example Script: Connecting to the devices in the testbed, checking if the IOS-XE version is compliant and disconnect:

    from pyats import aetest
from genie import testbed

class CommonSetup(aetest.CommonSetup):

    # Connecting to the devices in the testbed
    @aetest.subsection
    def connect(self, testbed):
        for device in testbed:
            device.connect()

class Testcase(aetest.Testcase):

    # Testcase to verify the devices are compliant in terms of software version
    @aetest.test
    def software_version(self, testbed, steps):
        for device_name in testbed.devices:
            device = testbed.devices[device_name]

            os = device.parse("show version")["version"]["xe_version"]

            with steps.start(f"Testing IOS-XE Version on device {device_name}") as step:
                if not os == "17.13.01a":
                   step.failed(f"OS is not compliant, IOS-XE version is {os}")   
                else:
                    step.passed("OS is compliant")

class CommonCleanup(aetest.CommonCleanup):

    # Disconnecting from the devices
    @aetest.subsection
    def disconnect(self,testbed):
        for device in testbed:
            device.disconnect()        

if __name__ == "__main__":
    devices = testbed.load("testbed.yml")
    aetest.main(testbed=devices)

  3. Job definition

    You could execute the previous script directly, but the superior way is to create a dedicated job file. This will run the pyATS tests and automatically generate a report about all the test cases. A simple job file could look like this:

    from pyats.easypy import run

def main(runtime):
    run(testscript="testscript.py", runtime=runtime)

Netbox dynamic inventory

In the previous example, we relied on a static testbed, which was a manually defined YAML file used to describe the network environment for our pyATS tests. While this approach works, it requires frequent updates as the network changes, making it inefficient and error-prone in dynamic environments.

To overcome these challenges, we will now integrate NetBox as a dynamic source for our pyATS testbed. By dynamically querying NetBox, we can ensure that our testbed always reflects the latest network state without requiring manual intervention.

For our specific use case, we want to filter devices that are in a planned state and belong to your pod.

Lets explore the additions we need to make to have a dynamic Netbox inventory.

First of all we need two additional imports:

  • NetBox: This import allows us to retrieve testbed data dynamically from NetBox. Instead of manually defining devices in a YAML file, we will fetch them directly from the NetBox API.
  • Genie: Once we retrieve the device information from NetBox, we need to load it into pyATS as a valid testbed. Genie provides the necessary functionality to handle this conversion.
from pyats.contrib.creators.netbox import Netbox
from genie import testbed

Now that we have the required imports, the next step is to instantiate the NetBox testbed. This object will query NetBox, apply our filter criteria, and generate a testbed dynamically.

nb_testbed = Netbox(
            netbox_url = os.getenv("NETBOX_API"),
            user_token = os.getenv("NETBOX_TOKEN"),
            def_user = "cli_user", 
            def_pass = "cli_password",
            url_filter="status=planned&tenant=pod1", # Filter devices that are in 'planned' state and belong to 'pod1'
            verify = False
        )

Once we have applied our filters and queried NetBox, we need to generate the testbed from the retrieved data.

tb = nb_testbed._generate()

As we have generated the testbed, the next step is to load it into pyATS so that it can be used in our test execution.

devices = testbed.load(tb)

Now that we have successfully generated and loaded the dynamic testbed, the final step is to pass it to the test script so that it can be used during execution. To do this, we specify the testbed as an argument in the run() function:

run(testscript="testscript.py", testbed=devices, runtime=runtime)

Step 1: Edit the job script to use Netbox as dynamic inventory

  1. Open the file job.py in the pyATS directory.

  2. Paste the following code into your job.py and make sure the pod variable is set to match your pod:

    from pyats.contrib.creators.netbox import Netbox
from pyats.easypy import run
from genie import testbed
import os

script_path = os.path.join(os.path.dirname(__file__), "test_network.py")

def main(runtime):

    netbox_url = os.getenv("NETBOX_API")
    netbox_token =  os.getenv("NETBOX_TOKEN")

    pod = "pod3"
    status = "planned"
    url_filter = f"status={status}&tenant={pod}"

    nb_testbed = Netbox(
        netbox_url = netbox_url,
        user_token = netbox_token,
        def_user = "netadmin",
        def_pass = "C1sco12345",
        url_filter=url_filter,
        verify = False
    )

    tb = nb_testbed._generate()
    devices = testbed.load(tb)

    run(testscript=script_path, testbed=devices, runtime=runtime, netbox_url=netbox_url, netbox_token=netbox_token)

Step 2: Review the test-script

  1. Open the file test_network.py in the pyATS directory.

  2. Let's check what we are doing in there:

    • CommonSetup: Our CommonSetup is handling the connection to the devices in the testbed
    • Testcases: We have one simple Testcase in place. We check if the right NTP server is configured and also synchronized.
    • CommonCleanup: Disconnecting from the device, and update the netbox status from planned to active if the NTP test was successful.

Step 3: Run the pyATS job and expect it to fail

Now we can run our pyATS job. At this stage we will expect it to fail because we did not onboard the switch by now. We will onboard the switch in the next Mission with a Gitlab CI/CD pipeline and then run the test to verify if the switch was deployed sucessfully.

  1. Run the script:

    pyats run job ./pyATS/job.py

  2. This should result in a connection attempt to your switch and will eventually fail because the switch still is in factory default configuration.

    Warning

    The test is expected to fail as the switch is in factory configuration and has no credentials configured for authentication

    Example failed test

  3. Make sure to commit and push your changes to your Git repository:

    git add .
git commit -m "mission-4"
git push