Working With Legacy Code: Extending Shiny Modules with the Sprout Technique and Test Driven Development
Making changes to a legacy system can be daunting. How can we develop code when we don’t have unit tests for a part of a system we need to change? How do we ensure a new feature is unit-tested when the rest of the codebase lacks them? One approach would be to just scrape the module off (or even the whole project!) and start from scratch, but it’s often <a href="https://understandlegacycode.com/blog/ship-of-theseus-avoid-rewrite-legacy-system/" target="_blank" rel="noopener">not worth the cost.</a> So what is the alternative? Use the <a href="https://understandlegacycode.com/blog/key-points-of-working-effectively-with-legacy-code/#1-the-sprout-technique" target="_blank" rel="noopener">Sprout Technique</a> with Test Driven Development. Table of Contents: <ul><li><a href="#task">The Task</a></li><li><a href="#sprout">Sprout Technique</a></li><li><a href="#insertion">Identify Insertion Point</a></li><li><a href="#isolation">Develop Code in Isolation</a></li><li><a href="#legacy">Call Your Code from the Legacy Code</a></li><li><a href="#summary">Summarizing Sprout Technique and Test Driven Development</a></li></ul> <hr /> <h2 id="task">The Task</h2> Let’s consider an app that displays names of tables from a data lake that are used to build visuals. It informs users from which version of data the plots are derived. This feature needs to be extended to show tables changes history to allow users to see previous versions of data, by whom and when they were changed. Let’s formulate this requirement in a form of <a href="https://resources.scrumalliance.org/Article/need-know-acceptance-criteria" target="_blank" rel="noopener">acceptance criteria</a>: <ul><li style="font-weight: 400;" aria-level="1">Data changes history is displayed in a table.</li><li style="font-weight: 400;" aria-level="1">Table is displayed in “Data” section</li></ul> Note that we don’t know yet how these criteria should be satisfied. Currently, we know only what needs to be delivered to meet the business goal. <h2 id="sprout">Sprout Technique</h2><ol><li style="font-weight: 400;" aria-level="1">Identify insertion point.</li><li style="font-weight: 400;" aria-level="1">Develop code in isolation.</li><li style="font-weight: 400;" aria-level="1">Call your code from the Legacy Code.</li></ol> <a href="https://understandlegacycode.com/blog/key-points-of-working-effectively-with-legacy-code/#1-the-sprout-technique" target="_blank" rel="noopener">The Sprout Technique</a> provides a structured approach to gradually introduce new code into legacy systems, ensuring smoother transitions and improved maintainability. <h2 id="insertion">Identify Insertion Point</h2> There is a module that already displays the names of tables used. We need to <a href="https://understandlegacycode.com/blog/key-points-of-working-effectively-with-legacy-code/#identify-seams-to-break-your-code-dependencies" target="_blank" rel="noopener">i</a>dentify the boundaries of the feature we're extending. In this case, it’s just a pair of <code>uiOutput</code> and <code>renderUI</code> in one legacy (untested) module that displays currently used tables names. This is the place where we will inject our new code. <pre><code> mod_data_ui <- function(id) { ns <- shiny::NS(id) shiny::div( shiny::h1("Data"), # ..., shiny::uiOutput(ns("data_summary")) ) } <br>mod_data_server <- function(input, output, session) { # ... output$data_summary <- renderUI({ # Some business logic }) } </code></pre> Learning about the code where you are injecting a new feature will help you understand what data is available there and how the new code can interact with its surroundings - it will help you drive the decision on what the new module’s interface can be. <h2 id="isolation">Develop Code in Isolation</h2> Let’s recall our criteria: <ul><li style="font-weight: 400;" aria-level="1">Data changes history is displayed in a table.</li><li style="font-weight: 400;" aria-level="1">Table is displayed in “Data” section</li></ul> From exploration of the existing code we see that an object called <code>StudyData</code> is available in the parent module that has a <code>get_data_history</code> method allowing you to fetch the data you need. <pre><code> StudyData <- R6::R6Class( classname = "StudyData", public = list( # Other methods # ... get_data_history = function() { # ... } ) ) </code></pre> Knowing that we can create an implementation list for the feature: <ul><li style="font-weight: 400;" aria-level="1">The changes history is fetched using the <code>StudyData</code> object.</li><li style="font-weight: 400;" aria-level="1">The changes history is displayed in the UI.</li></ul> Let’s set up our first test, we will use <a href="https://robertmarshall.dev/blog/arrange-act-and-assert-pattern-the-three-as-of-unit-testing/" target="_blank" rel="noopener">Arrange, Act, Assert</a> pattern to guide our thinking: <pre><code> describe("mod_data_history", { it("should fetch data history", { # Arrange # Act # Assert }) }) </code></pre> Let’s focus on the first item in our implementation list. Given that we need to use <code>StudyData</code> object there are two possibilities: <ul><li style="font-weight: 400;" aria-level="1">We can pass <code>StudyData</code> to the module.</li><li style="font-weight: 400;" aria-level="1">Or we can pass the history data we fetched in the parent module.</li><li style="font-weight: 400;" aria-level="1">Or we can pass the history data we fetched in the parent module.</li></ul> Let’s stick to the first option to keep gates open for this module to fetch some more data to display. Let’s call our new module <code>mod_data_history</code> . We expect the module should call the <code>study_data$get_data_history()</code> method. We need to mock a <code>StudyData</code> object to simulate its behavior, it allows us not to rely on actual implementation (which requires connection to the data lake). Since <code>StudyData</code> is a R6 class, we could mock an object with structure by hand, or we can create a reusable, simple routine that clones a R6 object: <pre><code> create_mock_r6_class <- function(generator) { checkmate::assert_class(generator, "R6ClassGenerator") structure( purrr::imap(generator$public_methods, mockery::mock), class = generator$classname ) } </code></pre> Then our mock object generator will look like: <pre><code> .create_mock_study_data <- function() { create_mock_r6_class(StudyData) } </code></pre> Note that we’re creating a wrapper for create_mock_r6_class(StudyData) - it may seem redundant now, but it’ll allow you to quickly substitute/extend this mock in all tests that use it. Then our test becomes: <pre><code> .create_mock_study_data <- function() { create_mock_r6_class(StudyData) } <br>describe("mod_data_history", { it("should fetch data history", { # Arrange study_data <- .create_mock_study_data() <br> # Act <br> # Assert }) }) </code></pre> For now, we only expect that this module will call this get_data_history method. We can use mockery::expect_called to check if this method has been called once. Let’s put that into the Assert block: <pre><code> describe("mod_data_history", { it("should fetch data history", { # Arrange study_data <- .create_mock_study_data() <br> shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act <br> # Assert mockery::expect_called( study_data$get_data_history, n = 1 ) } ) }) }) </code></pre> Running tests will throw errors 🔴 that <code>mod_data_history_server</code> object cannot be found. Let’s create the module: <pre><code> mod_data_history_ui <- function(id) { ns <- shiny::NS(id) <br>} <br>mod_data_history_server <- function(id, study_data) { shiny::moduleServer(id, function(input, output, session) { <br> }) } </code></pre> Now the test fails with an expected message 🔴, mock object has not been called yet: <pre><code> Failure (test-mod_data_history.R:12): mod_data_history: should fetch data history mock object has not been called 1 time </code></pre> To make the test pass we add a call to this method in the module: <pre><code> mod_data_history_ui <- function(id) { ns <- shiny::NS(id) } <br>mod_data_history_server <- function(id, study_data) { shiny::moduleServer(id, function(input, output, session) { data_history <- study_data$get_data_history() <br> }) } <br></code></pre> <pre><code> ✔ | F W S OK | Context ✔ | 1 | mod_data_history <br>══ Results ══════════════════════════════════════════════════════════════════════════════════════ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ] <br>🎯 Your tests hit the mark 🎯 </code></pre> 🟢 So far so good, we have met the first implementation requirement. <ul><li style="font-weight: 400;" aria-level="1">Changes history is fetched using the <code>StudyData</code> object.</li><li style="font-weight: 400;" aria-level="1">Changes history is displayed in the UI.</li></ul> Now we need to display the data that we just fetched. That means we need another test case for the second item in our list: <pre><code> it("should display data history", { # Arrange study_data <- .create_mock_study_data() <br> shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act <br> # Assert <br> } ) }) </code></pre> We don’t know yet what type of HTML should be produced to display this data, but from the server side we may expect that a method that parses history data and builds a HTML will be called. We will force <code>output$data_history</code> to evaluate and check if the display method has been called. Note that we only make one assumption - to which output slot we send the HTML – we don’t lock ourselves to a specific rendering engine with this test (e.g., <code>renderTable</code>, <code>renderUI</code>). <b>We’re not checking explicitly what value </b><code>output$data_history</code> yields. <pre><code> .study_data_history <- function() { tibble::tibble( data_name = c("data_1_v1", "data_1_v2", "data_1_v3"), user = c("user_1", "user_2", "user_1"), updated = c("2023-06-29 17:49:12", "2023-05-29 17:49:12", "2023-04-29 17:49:12"), size = c(1000, 1000, 1000) ) } <br>.create_mock_study_data <- function() { mock <- create_mock_r6_class(StudyData) mock$get_data_history <- mockery::mock(.study_data_history()) mock } <br># ... <br>it("should display data history", { # Arrange study_data <- .create_mock_study_data() mock_render_method <- mockery::mock() <br> shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act output$data_history <br> # Assert mockery::expect_args( mock_render_method, n = 1, data = .study_data_history() ) } ) }) </code></pre> You can see how we extended the mock in <code>.create_mock_study_data</code> to mock a return value from its method. This test fails 🔴 as expected – we don’t have an output yet: <pre><code> ✔ | F W S OK | Context ✖ | 1 1 | mod_data_history ───────────────────────────────────────────────────────────────────────────────────────────────── Error (test-mod_data_history.R:40): mod_data_history: should display data history Error in `.subset2(x, "impl")$getOutput(name)`: The test referenced an output that hasn't been defined yet: output$proxy1-data_history </code></pre> Let’s add outputs, we will use <code>{reactable}</code> as it implements all features we need to display the data in a shape we need. <pre><code> mod_data_history_ui <- function(id) { ns <- shiny::NS(id) reactable::reactableOutput(ns("data_history")) } <br>mod_data_history_server <- function(id, study_data) { shiny::moduleServer(id, function(input, output, session) { data_history <- study_data$get_data_history() <br> output$data_history <- reactable::renderReactable({ render_data_history(data_history) }) }) } </code></pre> Tests still fail 🔴, this time with an error: <pre><code> Error (test-mod_data_history.R:40): mod_data_history: should display data history Error in `render(data_history)`: could not find function "render_data_history" </code></pre> We need to either <a href="https://en.wikipedia.org/wiki/Method_stub" target="_blank" rel="noopener">stub</a> <code>render_data_history</code> it or inject it to the module. Let’s use stubbing for now, as we don’t need to parametrize this module with a rendering function: <pre><code> it("should display data history", { # Arrange study_data <- .create_mock_study_data() mock_render_method <- mockery::mock() mockery::stub(mod_data_history_server, "render_data_history", mock_render_method) <br> shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act output$data_history <br> # Assert mockery::expect_args( mock_render_method, n = 1, data = .study_data_history() ) } ) }) </code></pre> Test are green now 🟢. We fulfilled both points of our implementation plan: <ul><li style="font-weight: 400;" aria-level="1">Changes history is fetched using the <code>StudyData</code> object.</li><li style="font-weight: 400;" aria-level="1">Changes history is displayed in the UI.</li></ul> <pre><code> ✔ | F W S OK | Context ✔ | 4 | mod_data_history [0.4s] <br>══Results ══════════════════════════════════════════════════════════════════════════════════════ Duration: 0.4 s <br>[ FAIL 0 | WARN 0 | SKIP 0 | PASS 4 ] </code></pre> The whole test file looks like this: <pre><code> .study_data_history <- function() { tibble::tibble( data_name = c("data_1_v1", "data_1_v2", "data_1_v3"), user = c("user_1", "user_2", "user_1"), updated = c("2023-06-29 17:49:12", "2023-05-29 17:49:12", "2023-04-29 17:49:12"), size = c(1000, 1000, 1000) ) } <br>.create_mock_study_data <- function() { mock <- create_mock_r6_class(StudyData) mock$get_data_history <- mockery::mock(.study_data_history()) mock } <br>describe("mod_data_history", { it("should fetch data history", { # Arrange study_data <- .create_mock_study_data() shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act # Assert mockery::expect_called( study_data$get_data_history, n = 1 ) } ) }) it("should display data history", { # Arrange study_data <- .create_mock_study_data() mock_render_method <- mockery::mock() mockery::stub(mod_data_history_server, "render_data_history", mock_render_method) shiny::testServer( app = mod_data_history_server, args = list(study_data = study_data), { # Act output$data_history # Assert mockery::expect_args( mock_render_method, n = 1, data = .study_data_history() ) } ) }) }) </code></pre> Now we only need an implementation of the <code>render_data_history</code> function, the design of this function can also be driven by tests. Start from listing observable criteria – how we expect this function to behave. Add the first test case,<a href="https://www.oreilly.com/library/view/modern-c-programming/9781941222423/f_0054.html" target="_blank" rel="noopener"> red 🔴 → green 🟢 → refactor</a>. The function will accept data from <code>StudyData$get_data_history</code> and return a <code>reactable</code> object, since we chose this package for rendering the data. <h2 id="legacy">Call Your Code from the Legacy Code</h2> Once we implement <code>render_data_history</code> function, we can inject new module to the existing code: <pre><code> mod_data_ui <- function(id) { ns <- shiny::NS(id) shiny::div( shiny::h1("Data"), # ..., mod_data_history_ui(ns("data_summary")) ) } <br>mod_data_server <- function(input, output, session) { # ... mod_data_history_server("data_summary", study_data) } </code></pre> Now we can mark our acceptance criteria as done! <ul><li style="font-weight: 400;" aria-level="1">Data changes history is displayed in a table.</li><li style="font-weight: 400;" aria-level="1">Table is displayed in “Data” section</li></ul> <h2 id="summary">Summarizing Sprout Technique and Test Driven Development</h2> Using <a href="https://understandlegacycode.com/blog/key-points-of-working-effectively-with-legacy-code/#1-the-sprout-technique">Sprout Technique</a> and Test Driven Development, we’ve successfully injected a new module into a legacy one. Thanks to tests we have a documented characterization of this new module, it: <ul><li style="font-weight: 400;" aria-level="1">should fetch data history,</li><li style="font-weight: 400;" aria-level="1">should display data history.</li></ul> We have robust tests that expect a call to an established interface and expect a call to a rendering function. This test suite characterizes what this module does without knowing those functions' implementation details. Lower level details are covered by unit tests for both functions – it allows this module and those functions to evolve independently of each other. Tests for the module will remain valid when: <ul><li style="font-weight: 400;" aria-level="1">structure of data returned by <code>get_data_history</code>changes, e.g. when API changes,</li><li style="font-weight: 400;" aria-level="1">HTML returned by <code>render_data_history</code> changes, e.g. when implementing a new design or switching to a different tables' library.</li></ul> Start using TDD now to iterate faster and more confidently with legacy codebases!
