class: center, middle, inverse, title-slide # Developing Powerful Shiny Applications in an Enterprise Environment ## Best Practices and Lessons Learned ### Eric Nantz
Sr. Research Scientist @ Eli Lilly and Company
R/Pharma Conference | 16 August 2018
Slides:
rpodcast.gitlab.io/rpharma2018
@thercast
thercast
r-podcast.org
--- layout: true <div class="my-footer"><span>R/Pharma 2018              Developing Shiny Applications in Enterprise Environments              rpodcast.gitlab.io/rpharma2018</span></div> --- class: fullscreen, inverse, top, center, text-black background-image: url(img/long_road.jpg) .font250[**My Journey with Shiny**] --- # 2012 & 2013... * [`shiny`](http://shiny.rstudio.com/) released to CRAN and I became an early adopter * RStudio releases open-source __shiny server__ available for linux * Deployed on my home server and instantly hooked! -- 🤔 This could be a game-changer at work ... --  --  --  -- Recommended Reading: [Analytics Administration for R](https://rviews.rstudio.com/2017/06/21/analytics-administration-for-r/) by Nathan Stephens --- # Shiny catches on .... .pull-left[ .font150[ * Multiple statisticians & scientists begin creating Shiny applications * Difficult to keep up with the demands! + Licensed RStudio Server Professional and __Shiny Server Professional__ * Organized an internal __R & Shiny Day__ in partnership with RStudio ] ] .pull-right[  ] --- class: inverse, clear, center, middle # Practical Advice --- background-image: url(img/with-great-power-comes-great-responsibility.jpg) background-size: contain class: center, clear --- # Application Architecture For large-scale applications, I highly recommend using [__modules__](http://shiny.rstudio.com/articles/modules.html): -- - Set of R functions optimized for `shiny` applications, typically composed of user-interface and server-side logic - Avoid namespace collisions when using same widget across different parts of your app - Allows you to __compartmentalize__ distinct app components - 🚧 Passing values back and forth can be tricky! -- .pull-left[ .code80[ ```r schemeSelectUI <- function(id) { # mandatory call to set up unique namespace ns <- NS(id) uiOutput(ns("select_scheme_render")) } # within application UI schemeSelectUI(ns("first_select")) # within application server-side processing my_scheme <- callModule(schemeSelect, "first_select", scheme_choices) ``` ] ] .pull-right[ .code50[ ```r schemeSelect <- function(input, output, session, scheme_choices, label = "Scheme") { # render the select input based on the available schemes output$select_scheme_render <- renderUI({ ns <- session$ns selectInput( inputId = ns("select_scheme"), label = label, choices = scheme_choices(), selected = scheme_choices()[1], selectize = FALSE ) }) outputOptions(output, "select_scheme_render", suspendWhenHidden=FALSE) # add more related objects if needed reactive({ input$select_scheme}) } ``` ] ] --- background-image: url(img/picard_ui_meme.jpg) background-size: 800px background-position: 50% 50% # Enhancing your user interfaces --- # Enhancing your user interfaces Investing time to prevent cryptic errors based on user interactions will make your development __much easier__! -- <center></center> * Created by [Dean Attali](https://deanattali.com/) * Wraps powerful javascript utilities in easy-to-use functions + Toggle display of user interfaces elements + Disable and enable buttons or other inputs + Easily bind custom javascript code in your application .left-column-med[ .code70[ ```r # somewhere in app UI actionButton(ns("activate_session"), "Activate Session", class = "btn-primary") ``` ] ] .right-column-med[ .code70[ ```r # somewhere in server logic observe({ cond <- length(input$mytable_rows_selected) != 0 shinyjs::toggleState( id = "activate_session", condition = cond) }) ``` ] ] --- # Enhancing your user interfaces Investing time to prevent cryptic errors based on user interactions will make your development __much easier__! [__shinyFeedback__](https://github.com/merlinoa/shinyFeedback): Displaying user feedback next to Shiny inputs  .left-column-med[ .code70[ ```r # ui element textInput( inputId = "indication", label = "Indication", value = "" ) ``` ] ] .right-column-med[ .code75[ ```r # server-side processing observeEvent(input$indication, { shinyFeedback::feedbackWarning( inputId = "indication", condition = stringr::str_length(input$indication) > 18, text = "Slide title might be split to multiple lines, consider abbreviating" ) }) ``` ] ] --- class: inverse, center, middle, clear # Reproducibility & Automation --- # Personalized Experience Large-scale applications used extensively in __iterative__ workflows -- How can we let the user resume their work and initialize multiple workflows? -- * `shiny` now supports [__bookmarkable state__](http://shiny.rstudio.com/articles/bookmarking-state.html) (URL or server-side) + Useful for many workflows, but I envisioned more than just state of application -- 💡 Shiny Server Pro & RStudio Connect: Application processes owned by the __logged-in user__! .pull-left[ ```r user <- reactive({ if(!is.null(session$user)) { * my_user <- session$user } else { my_user <- Sys.getenv("USER") } return(my_user) }) ``` ] .pull-right[  ] --- # Application Workflows  -- * Create an overall application session directory: `/home/<logged_in_user>/.myapp_files` * Sync [`packrat`](https://rstudio.github.io/packrat/) application library, enabling a __reproducible__
execution environment * Allow user to create multiple __projects__ for separation of workflows -- .left-column-small[ .code60[ ``` /home/bob/.myapp_files -- hpc_workdir |__packrat -- mysession * |__archivist_repo |__backpack.db |__gallery * |__settings.RData -- sessions.sqlite3 ``` ] .code60[ ] ] .right-column-big[ .font90[ 📒 Inspired by the [Radiant](https://radiant-rstats.github.io/docs/) Shiny app created by Vincent Nijs 💡 [`archivist`](https://github.com/pbiecek/archivist) package allows you __version control__ R objects to disk! Recommended Reading: [Shiny + archivist = reproducible interactive exploration](http://smarterpoland.pl/index.php/2016/06/shiny-archivist-reproducible-interactive-exploration/) ] ] --- # Offload Processing .pull-left[ .font90[ When possible: Send CPU-intensive and __High Throughput Computing__ (HTC) workflows to a separate computing infrastructure * Less burden on the server * User can safely interact with other parts of application while jobs are processing Possible
based solutions: * [`batchtools`](https://mllg.github.io/batchtools/): Wraps HPC job submissions for variety of systems (Slurm, SGE, OpenLava, TORQUE/OpenPBS, LSF, Docker Swarm) * [`future.batchtools`](https://github.com/HenrikBengtsson/future.batchtools): utilizes the [`future`](https://github.com/HenrikBengtsson/future)
package API for parallel and distributed processing via `batchtools` * [`drake`](https://github.com/ropensci/drake):
-focused pipeline toolkit for reproducibility and HPC (Will Landau) * [`clustermq`](https://github.com/mschubert/clustermq): Utilizes `ZeroMQ` to send jobs to various HPC systems ] ] .pull-right[  .font80[ __Incorporating within `shiny`__: * [`reactiveValues`](http://shiny.rstudio.com/reference/shiny/1.0.5/reactiveValues.html) to track progress and key events * Inform user via [modals](http://shiny.rstudio.com/articles/modal-dialogs.html) or the new [`shinyalert`](https://daattali.com/shiny/shinyalert-demo/) package * Cache _integrated_ results to disk ] ] --- # Launching Simulations  -- .code60[ ```r observeEvent(input$launchSims, { # perform sanity checks of the user's settings check_the_settings() # launch simulations launch_sims(sessionDir = parent_dir, simID = hpcSimSelect(), n_batch = n_batch) # answer questions at this stage in process * any_sim_launched(TRUE) * which_sim_running(hpcSimSelect()) * total_jobs(n_batch) * total_sims(n_sims) * all_complete(FALSE) }) ``` ] --- # Monitoring _practically_ completed jobs .pull-left[ .code60[ ```r n_complete <- reactive({ # Re-execute after 5 seconds * invalidateLater(5000, session) # set default of 0 since nothing run yet n_completed_default <- 0 # perform checks only if sims launched if (any_sim_launched()) { # perform custom processing on results files n_completed_df <- get_n_completed(log_path) # if no sims have completed in the # launched project, then simply return 0 if (nrow(n_completed_df) < 1) { return(n_completed_default) } else { # update reactiveVal for sims_complete * sims_complete(nrow(n_completed)) return(nrow(n_completed)) } } else { return(n_completed_default) } }) ``` ] ] .pull-right[ Each job writes results to a file Once this file is available, I consider the job _practically_ completed `get_n_completed()` searches for all results files and returns data frame with status 💡 Even a typical `reactive` can be __invalidated__ at schedule intervals! ] --- # Reproducibility via RMarkdown Allowing user to download an __automated__ report with key results important for documentation and reproducibility -- .pull-left[ .code70[ ```r # within download handler processing # run knit_expand on reports report_dir <- "templates" report_files <- c("report1.Rmd", "report2.Rmd") filled_text <- rep(NA, length(report_files)) for (i in seq_along(report_files)) { filled_text[i] <- knitr::knit_expand( file = file.path(report_dir, report_files[i]), * user = user(), * sessionid = sessionid(), * sim_id = results_select() ) } cat(filled_text, file = file) ``` ] ] .pull-right[ .font80[ Template `report1.Rmd` YAML header: ] .code80[ ``` --- title: "Summary Report" output: html_document params: user: "{{user}}" sessionid: "{{sessionid}}" sim_id: "{{sim_id}}" --- ``` ] ] -- Helpful Resources * [Parameterized reports](https://bookdown.org/yihui/rmarkdown/parameterized-reports.html) chapter from __new__ [R Markdown: The Definitive Guide](https://bookdown.org/yihui/rmarkdown/) * `knitr::knit_expand()` [vignette](https://cran.r-project.org/web/packages/knitr/vignettes/knit_expand.html) --- background-image: url(img/end_is_near.png) background-size: 800px background-position: 50% 50% # Wrapping Up --- # Parting Thoughts .font110[ Shiny enables interactive analyses, reproducibility, and automation in prototyping __and__ production workflows ] -- .font110[ * 🛡 Ecosystem of packages available to __defend__ user interfaces from potential mishaps * 💼 Organizing souce code via __modules__ improves collaboration and organization * 📓 Ability to customize workflows tailored to iterative analyses, keeping __reproducibility__ front and center * 💻 Integration with external computing infrastructures to offload parallel analyses and obtain results faster ] -- .font110[ Additional Advice: * Use version control (i.e. `git` and GitHub/GitLab) for effective development, especially in team environments * Share prototypes with your intended audience to get feedback __early__ and __often!__ * Leverage the new [`shinytest`](https://rstudio.github.io/shinytest/articles/shinytest.html) package for automated testing * Have a robust design in mind for large-scale applications ] --- # Thank you! .center[ <table style="border-style:none;padding-top:30px;" class=".table"> <br /> <br /> <tr> <th style="padding-right:25px!important" align="center"><a href="https://twitter.com/thercast"> <i class="fab fa-twitter fa-3x"></i> </a></th> <th style="padding-left:25px!important" align="center"><a href="https://github.com/thercast"> <i class="fab fa-github fa-3x"></i> </a></th> <th style="padding-left:25px!important" align="center"><a href="https://gitlab.com/rpodcast"> <i class="fab fa-gitlab fa-3x"></i> </a></th> <th style="padding-left:25px!important" align="center"><a href="https://r-podcast.org"> <i class="fa fa-microphone fa-3x"></i> </a></th> </tr> <tr style="background-color:#fafafa"> <th style="padding-right:25px!important"><a href="https://twitter.com/thercast"> @thercast </a></th> <th style="padding-left:25px!important"><a href="https://github.com/thercast"> @thercast </a></th> <th style="padding-left:25px!important"><a href="https://gitlab.com/rpodcast"> @rpodcast </a></th> <th style="padding-left:25px!important"><a href="https://r-podcast.org"> r-podcast.org </a></th> </tr></table> ] .font110[ Other efforts in the
community: * Contributor to [R Weekly](https://rweekly.org/) * [RStudio Community](https://community.rstudio.com/) sustainer * Member of [Rbind](https://support.rbind.io/) administrator team Slides created with the [xaringan](https://slides.yihui.name/xaringan) package: [gitlab.com/rpodcast/rpharma2018](https://gitlab.com/rpodcast/rpharma2018) ]