Business needs to have dashboards validated. Many companies have dedicated Quality Assurance teams. As software engineers, we call the validation process testing. 

The process of validating an app by mimicking real users behavior is called end-to-end testing. It’s not some function, runs with some parameters, and we’re happy. It’s simulating real interactions with clicks and typing inputs in a programmatic way.

Such testing ensures that your applications (be it R/Shiny or Python or any language for that matter) work as they should, from the user interface down to the backend processes. Overall, this guarantees a reliable and seamless user experience and allows to tick one of the validation steps. 

‍

Explore the advantages of shinytest2 over Cypress for your Shiny app's E2E testing. Learn more in our blog post.

‍

When developing Shiny applications in Python, commonly known as Shiny for Python apps, ensuring robustness and reliability is important. While the R/Shiny ecosystem leverages tools like Cypress for end-to-end testing, Python introduces different tools and methods.

In this article, we dive into the world of end-to-end testing for Shiny for Python applications, utilizing the Pytest and Playwright libraries.

Why Playwright over Cypress?

Traditionally, R/Shiny and Rhino applications have utilized Cypress for end-to-end testing. Cypress is powerful but requires a Node/npm environment, which adds complexity to Python-centric projects. On the other hand, Playwright has a Python package that offers a Python-first experience, which aligns with the development environment of Shiny for Python apps.

Also, Shiny for Python already uses Playwright internally for end-to-end testing, suggesting a seamless integration within the ecosystem. While there is anticipation for Posit to introduce a dedicated testing API for Shiny for Python, leveraging Playwright remains the practical choice for now. 

The Essence of End-to-End Testing

End-to-end testing simulates real-user interactions with the application, such as clicking buttons, submitting forms, and navigating through pages. It is similar to manually exploring the app in a web browser but automated and more consistent. This testing method verifies that various components of the application work harmoniously and meet the specified requirements.

‍

Discover how to automate your testing process with Super Solutions for Shiny Apps #5. Visit this blog post to learn more.

Implementing Playwright in Shiny for Python

Playwright shines in its ability to mimic user actions with precision and reliability. One best practice in using Playwright is to select elements using the get_by_test_id method. This approach facilitates precise element selection, which is crucial for robust testing scripts.

Interestingly, Shiny applications automatically render data_testid attributes as data-testid, which Playwright uses. This is particularly handy since Python's syntax doesn't allow data-testid as a keyword argument directly, thus avoiding potential confusion and syntax errors.

‍

App Overview

Let’s demonstrate the practicalities of end-to-end (E2E) testing for a Shiny for Python application using Playwright. We focus on automating tests for common UI components and validating the application's functionality.

‍

The app has two cards. In one, we have two numeric inputs, a radio button, and a value box with the result. Depending on the radio button, the numeric inputs are added or subtracted. In the other card we have a slider and a value box. The value box shows the sum of all numbers between 0 and the slider value.

You can find the source code in our GitHub Repository. 

Testing Overview

To ensure our Shiny for Python app's robustness, we segment our testing strategy into different facets, each addressing specific components of the application. Let's delve into these:

Static Content Testing

This test module focuses on the app's static elements. It begins by verifying the presence and accuracy of static text within the app. 

Following this, the test ensures that the GitHub link, pointing to the repository with the source code, is present and correct. This check is crucial for maintaining the app's integrity and ensuring users have access to the correct resources.

‍Testing the Footer Text

The footer text is an essential element of the app, providing credit or important information. Here’s how we test it:

from playwright.sync_api import Page, expect

def test_footer(page: Page):
    # Given the app is open
    page.goto("http://localhost:8000")
    # Then the footer should contain the given text
    expect(page.get_by_test_id("footer")).to_contain_text("By Appsilon with 💙 ")

‍

In this function, test_footer, we direct Playwright to open the application running locally and check if the footer contains the expected phrase "By Appsilon with 💙". This is a straightforward test ensuring that the footer's static text remains as intended.

Verifying the GitHub Link

Next, we ensure the application includes a correct link to its GitHub source code repository:

def test_github_link(page: Page):
    # Given the app is open
    page.goto("http://localhost:8000")

    # When the user clicks on the source code on GitHub link
    locator = page.get_by_test_id("intro").get_by_text("source code on GitHub")
    with page.expect_popup() as page1_info:
        locator.click()
    page1 = page1_info.value

    # Then the user should be redirected to the correct page in a new tab
    expect(page1).to_have_url("https://github.com/Appsilon/pyshiny-e2e-playwright")

‍

This function, test_github_link, also starts by navigating to the app. It then identifies and clicks the link to the GitHub repository. The test asserts that this action opens a new tab with the expected URL, ensuring that users are directed to the correct GitHub page when they click on the link.

By including these specific tests, we ensure that our application not only looks right but also contains the correct references and information, maintaining a trustworthy and user-friendly experience. What’s important, when we make some changes to the codebase, we’re sure the functionality remains the same.

 Dynamic Content Testing

The calculation card is put under scrutiny here. This module leverages parametrized tests to assess the functionality of numeric inputs and radio buttons. 

By setting these elements to various values, the tests confirm the accuracy of the resulting calculations. Parametrization shines here, allowing multiple scenarios to be tested efficiently with the same test function, ensuring comprehensive coverage.

Parametrized testing is a powerful feature of pytest that allows us to run the same test function with different sets of data. In our case, we leverage this to assess the calculation functionality thoroughly:

import pytest
from playwright.sync_api import Page, expect

from pyshiny_e2e_playwright.calculation_type import CalculationType

@pytest.mark.parametrize(
    ("calculation_type", "first_operand", "second_operand", "expected"),
    [
        (CalculationType("Addition"), 2, 3, 5),
        (CalculationType("Addition"), 5, 3, 8),
        (CalculationType("Addition"), 0, 0, 0),
        (CalculationType("Subtraction"), 5, 3, 2),
        (CalculationType("Subtraction"), 3, 5, -2),
        (CalculationType("Subtraction"), 0, 0, 0),
    ],
)
def test_calculation(
    page: Page, calculation_type: CalculationType, first_operand: int, second_operand: int, expected: int
):
    # Given the app is open
    page.goto("http://localhost:8000")

    # When I select the calculation type
    calculation_radio = page.get_by_test_id("calculation_inputs").get_by_role("radio", name=calculation_type.value)
    calculation_radio.click()

    # And I fill the first and second operands
    first_number_input = page.get_by_test_id("calculation_inputs").get_by_label("Operand 1")
    first_number_input.fill(str(first_operand))

    second_number_input = page.get_by_test_id("calculation_inputs").get_by_label("Operand 2")
    second_number_input.fill(str(second_operand))

    # Then the result should be the expected value
    expect(page.get_by_test_id("calculation_result")).to_contain_text(str(expected))

In this test, we define a series of test cases for addition and subtraction, varying the operands and verifying that the application produces the correct result. The test function test_calculation automates the following steps:

1. Open the Application: It starts by navigating to the app running locally.
2. Select the Calculation Type: The test then chooses the type of arithmetic operation (Addition or Subtraction) using a radio button.
3. Input the Operands: It inputs the numbers for the operation into the designated fields.
4. Verify the Result: Finally, the test checks if the calculated result displayed on the page matches the expected value.

This approach not only ensures that different arithmetic scenarios are covered but also demonstrates how parametrized tests can make our testing suite more efficient and comprehensive. 

By automating these tests with Playwright and Pytest, we ensure that the application's calculation card functions correctly under various conditions, mirroring real-world usage.

Ready to improve your Shiny dashboards? Learn how through our video on effective user testing!

Dynamic Content Testing Through Javascript Interactions

In this segment, the slider card's functionality is evaluated. Tests dynamically adjust the slider to various positions, verifying the correct outcome for each. Special attention is paid to boundary conditions, guarding against common off-by-one errors like sum(range(n)). 

‍

Interestingly, this test involves executing JavaScript code within the browser to manipulate the slider’s position, a technique necessitated by the challenge of directly altering slider values in the DOM.

‍

Direct Interaction with the Slider through JavaScript

Unlike simple input fields, sliders represent a more complex UI element that requires interactive testing. Here’s how we tackle it in our tests:

import pytest
from playwright.sync_api import Page, expect

def set_slider_value_directly(page: Page, test_id: str, value: int):
    js_code = f"""
    var slider = $("[data-testid='{test_id}']").find(".js-range-slider");
    slider.data("ionRangeSlider").update({{ from: {value} }});
    """
    page.evaluate(js_code)

@pytest.mark.parametrize("slider_value", [25, 26, 50, 75])
def test_slider_directly(page: Page, slider_value: int):
    # Given the app is open
    page.goto("http://localhost:8000")

    # When I set the slider value to slider_value
    set_slider_value_directly(page, "range_sum_slider", slider_value)

    # Then the output should be the sum of numbers from 0 to slider_value
    expected = slider_value * (slider_value + 1) // 2
    expect(page.get_by_test_id("range_sum_result")).to_contain_text(str(expected))

‍

In this test, we utilize JavaScript within the Playwright environment to directly manipulate the slider's value, bypassing the typical user interaction flow. This method is essential because it allows precise control over the slider's position, which is especially useful for testing boundary conditions and ensuring that the sum calculation (a common operation tied to sliders) is accurate.

‍

Testing Boundary Conditions

To prevent common off-by-one errors, testing the slider at its boundary conditions is crucial:

@pytest.mark.parametrize(("slider_value", "expected"), [(0, 0), (100, 5050)])
def test_slider_boundaries(page: Page, slider_value: int, expected: int):
   # Given the app is open
   page.goto("http://localhost:8000")


   # When I set the slider value to slider_value
   set_slider_value_directly(page, "range_sum_slider", slider_value)


   # Then the output should be expected
   expect(page.get_by_test_id("range_sum_result")).to_contain_text(str(expected))

Here, we extend our testing to include the slider's minimum and maximum values, ensuring the calculation's accuracy across the entire range. This thorough testing approach mimics user interaction closely, despite being automated, thus ensuring the application behaves as expected under various scenarios.

While tools like Shinytest in RShiny offer direct manipulation of UI components, our current workflow with Playwright represents a practical interim solution, providing the necessary testing rigor for Python-based Shiny applications. This approach, albeit a workaround, is indicative of how automated testing can emulate user behavior, providing a comprehensive evaluation of the application’s interactive elements.

Conclusion

End-to-end testing is an invaluable part of the development process, especially for Shiny for Python applications. Although the ecosystem is still evolving, with potential dedicated testing tools on the horizon, Playwright stands out as a robust and compatible option for current needs.

For developers looking to deepen their understanding of unit testing with pytest, numerous resources and documentation are available. However, when it comes to end-to-end testing, integrating Playwright with Shiny for Python can lead to more reliable and user-friendly applications.

Embracing good practices in testing not only ensures the quality and performance of your applications but also enhances the development experience by providing a stable and predictable foundation for app functionality.

Take your Shiny applications to the next level with our expertise. Get in touch for a consultation.

FAQs 

• Can I integrate pytest with Playwright for testing my Shiny for Python applications?

Yes, pytest, with all its benefits, is integrated with Playwright here allowing you to combine end-to-end testing with parametrization, fixtures, mocking and more.  

• Why choose Playwright over Cypress for testing Shiny for Python applications?

Playwright is preferred for its Python-friendly approach, seamless integration with the Shiny for Python ecosystem, and because it doesn’t require a Node/npm environment, making it less complex for Python-centric projects.

‍