How to Use R in AWS Lambda
Cloud solutions rule the world of modern computing. Even the biggest players use solutions provided by Amazon (AWS stands for Amazon Web Services), Google or other cloud providers instead of establishing their own infrastructure. Such a solution saves time (and money), but the number of tasks that can be transferred outside is bigger than just using external servers. In fact, we can use various serverless solutions to deploy our application (e.g. <a href="https://cloud.google.com/appengine/">Google App Engine</a>), analyse real-time streaming data (e.g. <a href="https://aws.amazon.com/kinesis/">Amazon Kinesis</a>) and solve many other problems. <h2>What is AWS Lambda?</h2> Not only big tasks can be done without creating proper instances, buckets, containers, etc.; small, but frequent jobs can also be done without setting up an instance and keeping it “alive” waiting for requests. For such cases, Amazon created <a href="https://aws.amazon.com/lambda/">Lambda</a>. AWS Lambda is a serverless service for performing small (up to 15 minutes) tasks that can occur very frequently. Lambda can be triggered by almost any event performed on the AWS service (e.g. new data uploaded into <a href="https://aws.amazon.com/s3/">S3 Bucket</a>) and its result can be used in almost any AWS service (e.g. you can load results into <a href="https://aws.amazon.com/redshift/">Amazon Redshift</a> data warehouse). But the most <strong>impressive feature</strong> of AWS Lambda is that you do not have to care about the number of events! It will assign as many resources as needed and nothing more than that is required, so you will pay only for the actually used resources. And if you want just to test Lambda, Amazon provides you 1 million free requests and 400,000 GB-seconds of compute time per month. Lambda also has some restrictions. As was mentioned previously, the time for a single task is limited to 15 minutes. Also, memory is limited to 3 GB. Finally, AWS Lambda by default supports only a limited number of frameworks, including Java, Python, Go, Ruby, C#, Node.js and PowerShell. Fortunately, the latter is no longer an issue, since in December 2018, Amazon introduced <a href="https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/">custom runtimes for AWS Lambda</a>. This allows you to use almost any programming language, including R. <h2><strong>Use Case </strong></h2> Let’s say you have an ML model with hundreds of users, each one requesting predictions and uploading data. Of course you cannot just put the raw data inside the model, you have to pre-process it. You might have to normalize the values, and/or take care of missing data points. Using R for pre-processing is a good option. With an AWS Lambda + R combination, you don’t have to worry about the number of requests — each request is pre-processed separately and instantly, in parallel. It’s like going to the supermarket and every customer gets to walk straight to the counter — no waiting. And you don’t have to worry about creating and supporting the infrastructure, which means one less job to do for you. <h2>How to use R in AWS Lambda</h2> There are two ways to include your custom runtime (an environment with a framework you would like to use) in AWS Lambda. You can add it to your function code or you can provide it as a layer. We will focus on the latter. A layer is a ZIP archive, in which you can add all kinds of dependencies, including runtimes. You can provide up to 5 layers in your Lambda function, and, what is probably most important, you can reuse your layers in different Lambdas. <figure class="wp-block-image"><img class="wp-image-3524 td-animation-stack-type0-2" src="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?fit=640%2C158" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?w=2131 2131w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?resize=300%2C74 300w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?resize=768%2C189 768w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?resize=1024%2C252 1024w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?resize=640%2C158 640w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?resize=980%2C241 980w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?w=1280 1280w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?w=1920 1920w" alt="AWS Lambda, layers, execution environment diagram. " data-attachment-id="3524" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/lambda-diagram-02-01/" data-orig-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?fit=2131%2C525" data-orig-size="2131,525" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="Lambda-diagram-02-01" data-image-description="" data-medium-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?fit=300%2C74" data-large-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/Lambda-diagram-02-01.png?fit=640%2C158" /></figure> The key idea behind layers is that the Lambda mechanism looks for required dependencies starting from the first one in provided order. Such behaviour allows you to use separate layers with different parts of the environment required by your function. As a first layer, you should choose the one that contains the basic framework environment – in this particular case, we will need a layer with R (and all dependencies). If you need some additional packages, not included in the <strong>base</strong> layer, you can add them as an additional one. Sounds easy, right? <h2>Layers in R</h2> The biggest concern you may have now is how to get the required layers. Fortunately, reusability of layers extends beyond a single user sandbox. You can share your layers on various levels, allow access for a restricted group of people (e.g. your team) or make a layer public and accessible by the whole community. There are various custom runtimes for different frameworks that you can use in your project. At <a href="https://appsilon.com/">Appsilon Data Science Consulting</a>, we have also created layers that can be accessed by the community, and – guess what – they contain R. You can check the list of our R layers in the <a href="https://github.com/Appsilon/r-lambda-workflow">GitHub repository</a>. But how can you use them? It’s extremely simple and you can do it in a few steps <ul><li>First, create a new Lambda function. Remember to choose “custom runtime”!</li></ul> <figure class="wp-block-image"><img class="wp-image-3504 td-animation-stack-type0-2" src="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/custom_runtime.gif?zoom=1.5&fit=1303%2C355" alt="create a new Lambda function demo" width="640" height="175" data-attachment-id="3504" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/custom_runtime/" data-orig-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/custom_runtime.gif?fit=1303%2C355" data-orig-size="1303,355" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="custom_runtime" data-image-description="" data-medium-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/custom_runtime.gif?fit=300%2C82" data-large-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/custom_runtime.gif?fit=640%2C174" /></figure> <ul><li>Second, in the code editor, remove bootstrap and hello.sh files. Bootstrap file is one of the elements of the runtime and we will provide it in our R layer; hello.sh contains a Lambda “Hello world” function in the shell — but you want to use R, don’t you?</li><li>Third, create a new script file and name it. Remember to provide the ‘.R’ file extension! Put your R code inside. It must have the shape of a function!</li></ul> <figure class="wp-block-image"><img class="wp-image-3506 td-animation-stack-type0-2" src="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?fit=534%2C209" sizes="(max-width: 534px) 100vw, 534px" srcset="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?w=534 534w, https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?resize=300%2C117 300w" alt="sample Lambda function" data-attachment-id="3506" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/sample_function_lambda/" data-orig-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?fit=534%2C209" data-orig-size="534,209" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="sample_function_lambda" data-image-description="" data-medium-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?fit=300%2C117" data-large-file="https://i2.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/sample_function_lambda.png?fit=534%2C209" /></figure> <ul><li>Fourth, in the Handler field provide script name (without extension) followed by a dot and a function name (e.g. my_script.my_function).</li></ul> <figure class="wp-block-image"><img class="wp-image-3507 td-animation-stack-type0-2" src="https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?w=640" sizes="(max-width: 424px) 100vw, 424px" srcset="https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?w=424 424w, https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?resize=300%2C63 300w" alt="handler " width="424" height="89" data-attachment-id="3507" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/handler/" data-orig-file="https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?fit=424%2C89" data-orig-size="424,89" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="handler" data-image-description="" data-medium-file="https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?fit=300%2C63" data-large-file="https://i1.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/handler.png?fit=424%2C89" /></figure> <ul><li>Fifth, you will have to change your function timeout. The default of 3 seconds is too short for R.</li></ul> <figure class="wp-block-image"><img class="wp-image-3508 td-animation-stack-type0-2" src="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/timeout.gif?zoom=1.5&fit=811%2C363" alt="timeout " width="640" height="287" data-attachment-id="3508" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/timeout/" data-orig-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/timeout.gif?fit=811%2C363" data-orig-size="811,363" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="timeout" data-image-description="" data-medium-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/timeout.gif?fit=300%2C134" data-large-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/timeout.gif?fit=640%2C286" /></figure> <ul><li>Finally, add layers with an R runtime. Adding them is pretty simple if you know the ARN of a layer. For example,</li></ul> arn:aws:lambda:eu-central-1:599651763768:layer:basic-r:1 is an ARN of the base R layer created by <a href="https://appsilon.com/">Appsilon</a>, accessible in the eu-central-1 region (Frankfurt). Be aware that each layer is available only in a single region. You can provide it in the layers section. If you want to add more than one layer, remember that the one with the runtime has to be first! <figure class="wp-block-image"><img class="wp-image-3509 td-animation-stack-type0-2" src="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/add_layer.gif?zoom=1.5&fit=1600%2C692" alt="add a layer in R" width="640" height="277" data-attachment-id="3509" data-permalink="http://enhancedatascience.com/2019/07/09/how-to-use-r-in-aws-lambda/add_layer/" data-orig-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/add_layer.gif?fit=1600%2C692" data-orig-size="1600,692" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="add_layer" data-image-description="" data-medium-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/add_layer.gif?fit=300%2C130" data-large-file="https://i0.wp.com/enhancedatascience.com/wp-content/uploads/2019/07/add_layer.gif?fit=640%2C277" /></figure> That’s it. Congratulations, you have created an R function in AWS Lambda. To be sure that it works properly, you can use the included test tool. Just provide the proper function arguments in a JSON format. <h2>Create Your Own Environment</h2> Building your own environment for a custom language like R takes several steps and may be quite difficult, especially if that it is not a part of everyday routine. Just to make our lives easier, at <a href="https://appsilon.com/">Appsilon </a>we decided to create a unified workflow for that, and we want to share it with the community. With the workflow you can easily create your own runtime with your choice of R version and included packages. How to do that? You can find detailed instructions in our <a href="https://github.com/Appsilon/r-lambda-workflow">repository</a>, but here you can get a general view. The only thing that you have to do is <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html">configure your credentials in AWS CLI tool</a> and create a cryptographic key to be able to connect with the EC2 instances used in the process of preparing layers. After that, our scripts will do the work for you, and you will only have to publish your layers. To build custom runtime in AWS Lambda, you have to use a specific <a href="https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2">instance image</a>, but don’t worry, our workflow will choose it for you. If you want to create a base R layer, just provide a path to your key and the script will set up an instance, install R and download an archive with R installation. Another script will add some additional required files (remember the bootstrap file?) and the last thing to do will be just to publish the layer. Creating a layer with R packages is even more simplified. To do that you don’t need to install R because we have done that for you! Just use our AMI image with preinstalled R (AMI id: ami-0a1147e8e86aa6175) and add your packages. How to do that? Again, provide your AMI ID and names of the packages to our script and it will do the work for you. It will create the instance, install packages and download the archive. And again, just publish it as a layer. Of course, you may want to create your own AMI with a specific version of R, but you probably will not be surprised that you can do it with our scripts.
.png)
.png)