Developing Powerful Shiny Applications in an Enterprise Environment

# Developing Powerful Shiny Applications in an Enterprise Environment
## Best Practices and Lessons Learned
### Eric Nantz Sr. Research Scientist @ Eli Lilly and Company R/Pharma Conference | 16 August 2018 Slides: <a href='http://rpodcast.gitlab.io/rpharma2018/'>rpodcast.gitlab.io/rpharma2018</a> <a href='https://twitter.com/thercast'> @thercast</a> <a href='https://github.com/thercast'> thercast</a> <a href='https://r-podcast.org'>r-podcast.org</a>

---

<div class="my-footer">R/Pharma 2018&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;Developing Shiny Applications in Enterprise Environments&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;rpodcast.gitlab.io/rpharma2018</div>

---
class: fullscreen, inverse, top, center, text-black

---

# 2012 & 2013...

* [`shiny`](http://shiny.rstudio.com/) released to CRAN and I became an early adopter
* RStudio releases open-source __shiny server__ available for linux
* Deployed on my home server and instantly hooked!

🤔 This could be a game-changer at work ...

![:scale 30%](img/hat_rlogo.png)
--
![:scale 30%](img/hat_tux.png)
--
![:scale 30%](img/monitor_jericho.png)
--

Recommended Reading: [Analytics Administration for R](https://rviews.rstudio.com/2017/06/21/analytics-administration-for-r/) by Nathan Stephens

---

# Shiny catches on ....

.pull-left[
.font150[
* Multiple statisticians & scientists begin creating Shiny applications
* Difficult to keep up with the demands!
     + Licensed RStudio Server Professional and __Shiny Server Professional__
* Organized an internal __R & Shiny Day__ in partnership with RStudio
]
]

---

# Practical Advice

---
background-image: url(img/with-great-power-comes-great-responsibility.jpg)
background-size: contain
class: center, clear

---

# Application Architecture

For large-scale applications, I highly recommend using [__modules__](http://shiny.rstudio.com/articles/modules.html):
--

- Set of R functions optimized for `shiny` applications, typically composed of user-interface and server-side logic
- Avoid namespace collisions when using same widget across different parts of your app
- Allows you to  __compartmentalize__ distinct app components
- 🚧 Passing values back and forth can be tricky!

.pull-left[
.code80[
```r
schemeSelectUI <- function(id) {
 # mandatory call to set up unique namespace
 ns <- NS(id)
 uiOutput(ns("select_scheme_render"))
}

# within application UI
schemeSelectUI(ns("first_select"))

# within application server-side processing
my_scheme <- callModule(schemeSelect, 
 "first_select", 
 scheme_choices)
```
]
]

.pull-right[
.code50[
```r
schemeSelect <- function(input, output, session, 
 scheme_choices, label = "Scheme") {
 
 # render the select input based on the available schemes
 output$select_scheme_render <- renderUI({
 ns <- session$ns
 selectInput(
 inputId = ns("select_scheme"),
 label = label,
 choices = scheme_choices(),
 selected = scheme_choices()[1],
 selectize = FALSE
 )
 })
 
 outputOptions(output,
 "select_scheme_render", 
 suspendWhenHidden=FALSE)
 # add more related objects if needed
 reactive({ input$select_scheme})
}
```
]
]

---

background-image: url(img/picard_ui_meme.jpg)
background-size: 800px
background-position: 50% 50%

# Enhancing your user interfaces

---

# Enhancing your user interfaces

Investing time to prevent cryptic errors based on user interactions will make your development __much easier__!

<center>![:scale 20%](img/shinyjs-logo-whitebg-small.png)</center>

* Created by [Dean Attali](https://deanattali.com/)
* Wraps powerful javascript utilities in easy-to-use functions
    + Toggle display of user interfaces elements 
    + Disable and enable buttons or other inputs
    + Easily bind custom javascript code in your application

.left-column-med[
.code70[
```r
# somewhere in app UI
actionButton(ns("activate_session"), 
             "Activate Session", 
             class = "btn-primary")
```
]
]

.right-column-med[
.code70[
```r
# somewhere in server logic
observe({
 cond <- length(input$mytable_rows_selected) != 0
 shinyjs::toggleState(
 id = "activate_session",
 condition = cond)
})
```
]
]

---

# Enhancing your user interfaces

Investing time to prevent cryptic errors based on user interactions will make your development __much easier__!

[__shinyFeedback__](https://github.com/merlinoa/shinyFeedback): Displaying user feedback next to Shiny inputs

![](img/shinyfeedback.gif)

.left-column-med[
.code70[
```r
# ui element
textInput(
  inputId = "indication",
  label = "Indication",
  value = ""
)
```
]
]

.right-column-med[
.code75[
```r
# server-side processing
observeEvent(input$indication, {
  shinyFeedback::feedbackWarning(
    inputId = "indication",
    condition = stringr::str_length(input$indication) > 18,
    text = "Slide title might be split to multiple lines, 
            consider abbreviating"
  )
})
```
]
]

---

# Reproducibility & Automation

---

# Personalized Experience

Large-scale applications used extensively in __iterative__ workflows

How can we let the user resume their work and initialize multiple workflows?

* `shiny` now supports [__bookmarkable state__](http://shiny.rstudio.com/articles/bookmarking-state.html) (URL or server-side)
    + Useful for many workflows, but I envisioned more than just state of application

💡 Shiny Server Pro & RStudio Connect: Application processes owned by the __logged-in user__!

```r
user <- reactive({
 if(!is.null(session$user)) {
* my_user <- session$user
 } else {
 my_user <- Sys.getenv("USER")
 }
 return(my_user)
})
```
]

---

# Application Workflows

![:scale 80%](img/session_save_screenshot.png)

* Create an overall application session directory: `/home/<logged_in_user>/.myapp_files`
* Sync [`packrat`](https://rstudio.github.io/packrat/) application library, enabling a __reproducible__ execution environment
* Allow user to create multiple __projects__ for separation of workflows

]
]

.right-column-big[
.font90[
📒 Inspired by the [Radiant](https://radiant-rstats.github.io/docs/) Shiny app created by Vincent Nijs

💡 [`archivist`](https://github.com/pbiecek/archivist) package allows you __version control__ R objects to disk!

Recommended Reading: [Shiny + archivist = reproducible interactive exploration](http://smarterpoland.pl/index.php/2016/06/shiny-archivist-reproducible-interactive-exploration/)
]
]

---

# Offload Processing

.pull-left[
.font90[
When possible: Send CPU-intensive and __High Throughput Computing__ (HTC) workflows to a separate computing infrastructure

* Less burden on the server 
* User can safely interact with other parts of application while jobs are processing

Possible based solutions:

* [`batchtools`](https://mllg.github.io/batchtools/): Wraps HPC job submissions for variety of systems (Slurm, SGE, OpenLava, TORQUE/OpenPBS, LSF, Docker Swarm)
* [`future.batchtools`](https://github.com/HenrikBengtsson/future.batchtools): utilizes the [`future`](https://github.com/HenrikBengtsson/future) package API for parallel and distributed processing via `batchtools`
* [`drake`](https://github.com/ropensci/drake): -focused pipeline toolkit for reproducibility and HPC (Will Landau)
* [`clustermq`](https://github.com/mschubert/clustermq): Utilizes `ZeroMQ` to send jobs to various HPC systems
]
]

* [`reactiveValues`](http://shiny.rstudio.com/reference/shiny/1.0.5/reactiveValues.html) to track progress and key events
* Inform user via [modals](http://shiny.rstudio.com/articles/modal-dialogs.html) or the new [`shinyalert`](https://daattali.com/shiny/shinyalert-demo/) package
* Cache _integrated_ results to disk
]
]

---

# Launching Simulations

![:scale 60%](img/launch_sim_table.png)
--

```r
observeEvent(input$launchSims, {
  # perform sanity checks of the user's settings
  check_the_settings()
  
  # launch simulations
  launch_sims(sessionDir = parent_dir, simID = hpcSimSelect(), n_batch = n_batch)
  
  # answer questions at this stage in process
* any_sim_launched(TRUE)
* which_sim_running(hpcSimSelect())
* total_jobs(n_batch)
* total_sims(n_sims)
* all_complete(FALSE)
})
```
]

---

# Monitoring _practically_ completed jobs

```r
n_complete <- reactive({
 
 # Re-execute after 5 seconds
* invalidateLater(5000, session)
 
 # set default of 0 since nothing run yet
 n_completed_default <- 0
 
 # perform checks only if sims launched
 if (any_sim_launched()) {
 
 # perform custom processing on results files
 n_completed_df <- get_n_completed(log_path)
 
 # if no sims have completed in the 
 # launched project, then simply return 0
 if (nrow(n_completed_df) < 1) {
 return(n_completed_default)
 } else {
 # update reactiveVal for sims_complete
* sims_complete(nrow(n_completed))
 return(nrow(n_completed))
 }
 } else {
 return(n_completed_default)
 }
})
```

]
]

Once this file is available, I consider the job _practically_ completed

`get_n_completed()` searches for all results files and returns data frame with status

💡 Even a typical `reactive` can be __invalidated__ at schedule intervals!
]

---

# Reproducibility via RMarkdown

Allowing user to download an __automated__ report with key results important for documentation and reproducibility

```r
# within download handler processing
# run knit_expand on reports
report_dir <- "templates"
report_files <- c("report1.Rmd", "report2.Rmd")

filled_text <- rep(NA, length(report_files))

for (i in seq_along(report_files)) {
 filled_text[i] <- knitr::knit_expand(
 file = file.path(report_dir, report_files[i]), 
* user = user(),
* sessionid = sessionid(),
* sim_id = results_select()
 )
}

cat(filled_text, file = file)
```
]
]

.code80[
```
---
title: "Summary Report"
output: html_document
params:
  user: "{{user}}" 
  sessionid: "{{sessionid}}" 
  sim_id: "{{sim_id}}" 
---
```
]
]

Helpful Resources

* [Parameterized reports](https://bookdown.org/yihui/rmarkdown/parameterized-reports.html) chapter from __new__ [R Markdown: The Definitive Guide](https://bookdown.org/yihui/rmarkdown/) 
* `knitr::knit_expand()` [vignette](https://cran.r-project.org/web/packages/knitr/vignettes/knit_expand.html)

---

background-image: url(img/end_is_near.png)
background-size: 800px
background-position: 50% 50%

# Wrapping Up

---

# Parting Thoughts

.font110[
Shiny enables interactive analyses, reproducibility, and automation in prototyping __and__ production workflows
]

.font110[
* 🛡 Ecosystem of packages available to __defend__ user interfaces from potential mishaps
* 💼 Organizing souce code via __modules__ improves collaboration and organization
* 📓 Ability to customize workflows tailored to iterative analyses, keeping __reproducibility__ front and center
* 💻 Integration with external computing infrastructures to offload parallel analyses and obtain results faster
]

* Use version control (i.e. `git` and GitHub/GitLab) for effective development, especially in team environments
* Share prototypes with your intended audience to get feedback __early__ and __often!__
* Leverage the new [`shinytest`](https://rstudio.github.io/shinytest/articles/shinytest.html) package for automated testing
* Have a robust design in mind for large-scale applications
]

---

# Thank you!

.center[
<table style="border-style:none;padding-top:30px;" class=".table">
 
 
 <tr>
 <th style="padding-right:25px!important" align="center"><a href="https://twitter.com/thercast"> </a></th>
 <th style="padding-left:25px!important" align="center"><a href="https://github.com/thercast"> </a></th>
 <th style="padding-left:25px!important" align="center"><a href="https://gitlab.com/rpodcast"> </a></th>
 <th style="padding-left:25px!important" align="center"><a href="https://r-podcast.org"> </a></th>
 </tr>
 <tr style="background-color:#fafafa">
 <th style="padding-right:25px!important"><a href="https://twitter.com/thercast"> @thercast </a></th>
 <th style="padding-left:25px!important"><a href="https://github.com/thercast"> @thercast </a></th>
 <th style="padding-left:25px!important"><a href="https://gitlab.com/rpodcast"> @rpodcast </a></th>
 <th style="padding-left:25px!important"><a href="https://r-podcast.org"> r-podcast.org </a></th>
 </tr></table>
]

.font110[
Other efforts in the community:
* Contributor to [R Weekly](https://rweekly.org/)
* [RStudio Community](https://community.rstudio.com/) sustainer
* Member of [Rbind](https://support.rbind.io/) administrator team

Slides created with the [xaringan](https://slides.yihui.name/xaringan) package: [gitlab.com/rpodcast/rpharma2018](https://gitlab.com/rpodcast/rpharma2018)
]