Dashboard Framework Part 2: Running Shiny in AWS Fargate with CDK

Quantargo

Contributor:
Quantargo
Visit: Quantargo

In the previous post we outlined the architecture of a dashboard framework to run dashboards based on multiple technologies including Shiny and Flask in production. We will now show how to run a basic Shiny dashboard in AWS Fargate behind an Application Load Balancer in less than 60 lines of CDK code. To define our stack in a reproducible manner we will make use of the Amazon Cloud Development Kit (CDK) with Typescript. Starting from a basic CDK stack we now specify the most important components of our stack:

  1. The Application Load Balancer (ALB) to route traffic to our dashboards.
  2. The Fargate cluster to run our dashboard tasks in a scalable manner.

The deployed stack will finally run an example Shiny dashboard behind an Application Load Balancer. Note that the resulting stack will only run one dashboard without encryption. We’ll implement these features as part of the next post. The resulting CDK code can also be downloaded from Github at https://github.com/quantargo/dashboards.

Prerequisites

To run the following code examples make sure to have

  1. an AWS Account
  2. a locally configured AWS account by running e.g. aws configure with the aws CLI
  3. a local Node.js installation (version >= 14.15.0)
  4. Typescript: npm -g install typescript
  5. CDK (version >= 2.0): npm install -g aws-cdk

Initialize CDK and Deploy first App

To initialize a sample project we first create a project folder and within the folder execute cdk init:

> mkdir dashboards
  cd dashboards
  cdk init app --language typescript

This command creates a new CDK Typescript project and installs all required packages. The following 2 files are relevant for stack development:

  • bin/dashboards.ts: Main file which initializes CDK stack class. You can explicitly set the environment env if you use a different account or region for deployment.
  • lib/dashboards-stack.ts: CDK Stack class to which all components of our stack will be added.

Specify Application Load Balancer (ALB)

Next, we need to create an Application Load Balancer (ALB) within a new VPC which is responsible for secure connections and routing. We create a new VPC and add an internetFacing load balancer to it. This means that the load balancer will be accessible from the public internet and will therefore be placed into a public subnet. Within the lib/dashboards-stack.tsfile we put the following lines:

> // Put imports on top of the file
  import * as ec2 from 'aws-cdk-lib/aws-ec2';
  import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2'

  // Put below lines within the DashboardsStack constructor
  const vpc = new ec2.Vpc(this, 'MyVpc');

  const lb = new elbv2.ApplicationLoadBalancer(this, 'LB', {
    vpc: vpc,
    internetFacing: true,
    loadBalancerName: 'DashboardBalancer'
  });

Specify Dashboard Cluster

Next, we need to add an ECS cluster to our VPC to run our dashboards efficiently:

> // Put imports on top of the file
  import * as ecs from 'aws-cdk-lib/aws-ecs'

  // Put below lines within the DashboardsStack constructor
  const cluster = new ecs.Cluster(this, 'DashboardCluster', {
    vpc: vpc
  });

Add First Fargate Task Definition

We can now add our first Fargate dashboard to the cluster by specifying a task definition. We use the rocker/shiny Docker container as an example running on port 3838. This also requires respective port mappings in the container definition. Additionally, we use half a virtual CPU (512)—1024would equal a full one—and a memory size of 1024 MiB. By specifying the Fargate service we are already finished with the specification to run our first container in the cluster:

> const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDefinition', {
   cpu: 512,
   memoryLimitMiB: 1024,
  });

  const port = 3838

  const container = taskDefinition.addContainer('Container', {
    image: ecs.ContainerImage.fromRegistry('rocker/shiny'),
    portMappings: [{ containerPort: port }],
  })

  const service = new ecs.FargateService(this, 'FargateService', {
    cluster: cluster,
    taskDefinition: taskDefinition,
    desiredCount: 1,
    serviceName: 'FargateService'
  })

Put Service Behind ALB

Next, we put the Fargate service into an ALB target group so that traffic can be routed through the ALB:

> const tg1 = new elbv2.ApplicationTargetGroup(this, 'TargetGroup', {
   vpc: vpc,
   targets: [service],
   protocol: elbv2.ApplicationProtocol.HTTP,
   stickinessCookieDuration: cdk.Duration.days(1),
   port: port,
   healthCheck: {
     path: '/',
     port: `${port}`
   }
 })

Note that we added 2 parameters to the ALB target group definition:

  • stickinessCookieDuration: Since Shiny sessions are stateful we need to prevent the ALB to switch instances (in case there are more) during a session. The session duration set to one day should be sufficient.
  • healthCheck: The health check needs to specify the port (as string) and set to the container port 3838, as well.

Finally, we add an HTTP listener which directly forwards all incoming traffic to our dashboard:

> const listener = lb.addListener(`HTTPListener`, {
   port: 80,
   defaultAction: elbv2.ListenerAction.forward([tg1]) 
 })

Deploy

Before deployment you should also bootstrap your CDK environment:

> cdk bootstrap

Now the stack should be ready for deployment. As an extra step, you can now check if the stack can be successfully synthesized using

> cdk synth

Any errors popping up during cdk synth need to be fixed immediately. By continously using cdk synth we make sure that the feedback cycles during development are as short as possible. If cdk synth is successful we can now run

> cdk deploy

Finally, you should see the successful output message including the DashboardsStack.LoadBalancerDNSName which you can directly access through the browser:

> Outputs:
  DashboardsStack.LoadBalancerDNSName = DashboardBalancer-<9-DIGIT-NUMBER>.<region>.elb.amazonaws.com
  Stack ARN:
  arn:aws:cloudformation:<region>:<AWS-ACCOUNT-NO>:stack/DashboardsStack/<uuid>

    Total time: 297.67s

Destroy

If you don’t use the stack any more and to reduce cloud costs just run:

> cdk destroy

Visit Quantargo for additional insight on this topic: https://www.quantargo.com/blog/2022-03-11-creating-dashboard-framework-with-aws-part2.

Disclosure: Interactive Brokers

Information posted on IBKR Traders’ Insight that is provided by third-parties and not by Interactive Brokers does NOT constitute a recommendation by Interactive Brokers that you should contract for the services of that third party. Third-party participants who contribute to IBKR Traders’ Insight are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from Quantargo and is being posted with permission from Quantargo. The views expressed in this material are solely those of the author and/or Quantargo and IBKR is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

In accordance with EU regulation: The statements in this document shall not be considered as an objective or independent explanation of the matters. Please note that this document (a) has not been prepared in accordance with legal requirements designed to promote the independence of investment research, and (b) is not subject to any prohibition on dealing ahead of the dissemination or publication of investment research.

Any trading symbols displayed are for illustrative purposes only and are not intended to portray recommendations.