Eugene Istrati

Proud Father. Lucky Husband. DevOps | Automation | Serverless @MitocGroup. Former @AWScloud and @HearstCorp.

Terraform for Serverless Series: Enhanced Management of AWS Lambda Functions

September 19th, 2018 / 5 min read

I quickly became fascinated by terraform's simplicity and ease of use. As an engineer, you can tell when a tool just works. This reliability and trust in positive results makes it indispensable. And, sometimes, these experiences evolve into something beyond original capabilities. In this article, I would like to describe one of those experiences. Specifically, I will share our simplified version of compiling and deploying AWS Lambda functions using terraform.

Terraform for Serverless Applications

Serverless Applications with AWS Lambda and API Gateway is a very good piece of documentation and I quickly came to appreciate it. Although, in my humble opinion, I believe that most of us have more complex use cases that can't fit so easily into below steps (quoted from above documentation):


 $ cd example
 $ zip ../example.zip main.js
   adding: main.js (deflated 33%)
 $ cd ..
        

Although technically we can achive above situation by bundling our code with tools like webpack, in this article I would like to cover a larger pool of use cases by providing a better solution and more sustainable alternative that will work with terraform.

Terraform for AWS Lambda

My personal preference is to separate terraform configurations into group / type / service / function specific .tf files. So, normally, I would define my providers in provider.tf, setup my data in data.tf, specify my variables in variables.tf, configure my resources in main.tf and so on. But, for the purpose of this article, I'm putting everything all together into one single .tf file, ignoring best practices as long as it works:

Above terraform configuration assumes s3://terraform-serverless-example/v1.0.0/example.zip exists and is publicly accessible. In order to update this lambda function, just create and upload the new version (e.g. 1.0.1) of .zip file into S3 and execute terraform apply with new app version number (e.g. terraform apply -var="app_version=1.0.1").

Next, I would like to describe a better approach that is focused on AWS Lambda function's .zip file and terraform data configuration.

Terraform for ZIP File

Native terraform implementation for AWS Lambda doesn't provide any simple capability to create or update .zip files. But that didn't stop our team to come up with fully enhanced and fully automated process to build and compress everything we want related to AWS Lambda function into .zip file using native terraform features. Here below is a simplified example of what we've been able to achieve so far (which, by the way, I know for a fact that could be improved):

Although it's not mandatory, we have included in output's configuration ".*". This syntax will ensure that no matter what is the count number in resource, output will always provide some kind of value back.

Next, we have added to lambda function arguments like count to make sure that function is not created if program fails and source_code_hash to allow terraform natively compare if newly created .zip file is different from the one already deployed.

We have defined null data source that links to source_code_hash and depends on external data (described in next paragraph). This piece of code is triggered on every execution of terraform plan (as well as apply or destroy if plan is not passed as input variable).

Last, but not the least, we have defined external data that links to count. This piece of code is triggered on each execution of null data source. The "query" in external data allows us to pass any value or variable we need from terraform to the program. As result, terraform expects from the program to return a valid json.

In our terraform example from above, we call node as external program and build.js as the argument. Here below is the simplified version of build.js script:

To summarize, our build script implements 4 steps to achieve its goal:

  1. Installs production dependencies / node_modules using npm
  2. Creates bundled JavaScript using webpack
  3. Compresses bundled JavaScript (and JSON) files into .zip
  4. Uploads .zip file into S3

We have chosen Node.js for build.js only because the engineer who created this script feels more comfortable with JavaScript. But it's important to point out that it could have been build.sh or build.py (or literally any other language that can read input data from stdin and return json to stdout).

Still Needs Improvements

Unfortunately, above code is not perfect. We are aware of the following issues:

  • build.js will be executed at every terraform plan action (as well as apply or destroy if plan is not passed as input variable); we are working to optimize it and improve it by checking if any file in lambda_path was recently changed in comparison with timestamp of latest .zip file
  • source_code_hash is different for every execution of terraform plan and therefore tries to redeploy the lambda function on every execution of terraform apply; we are working to optimize it and improve it by persisting the latest .zip file and updating it with newly created .js (and .json) files which, in return, generates the same hash if no files have been changed

Spoiler Alert: all steps and work-arounds described in this article are carefully crafted into language agnostic functionality that will be released soon as a new feature in our open source project TerraHub CLI.

We would love to hear thoughts and comments on what could be done better.

Final Thoughts

TerraHub.io is the DevOps Hub for Terraform Automation. We provide managed services that simplify cloud resources management using terraform. If this is of your interest and you’d like to learn more, please feel free to reach out over Email, Twitter or LinkedIn. We’d be happy to help!