Getting Information Outside of Cloudformation

Infrastructure as code is great ideal, but in real world it is often not possible reach. Ideally all your cloud infrastructure would be sharing the single state, and allow references to any resource needed. In more realistic scenario your stack is one of many and some of the resources aren’t managed in code at all (or are created with another IaaC tool). Referencing to resources, or data, outside of the stack, hasn’t never been the strongest point of Cloudformation.

You can pass information between stacks with export/import, nested stacks, SSM parameter store, stack sets or sometimes even with copy-pasting. But what has been missing is the ability to reference external resources the same way as Terraform data sources do.

Extending Cloudformation

It has been possible to extend Cloudformation with custom resources for a long time. And with SAM templates, it became possible to combine the infrastructure and logic of custom resource into single template, assuming your code was compact enough to fit into template and didn’t depend on libraries outside of standard lambda runtimes.

While resource types are solving the same problem of extending Cloudformation, they are real 1st class citizens and comparable to any AWS provided resources. Major difference is you are responsible of running custom resources (e.g. Lambda-function), while resources types are run by AWS as part of Cloudformation service. This combined with Cloudformation Registry makes it much easier to share and consume private resource types, than custom resources, across multiple projects.

See Managing resources using AWS CloudFormation Resource Types for an overview of both models and their differences.

Nordcloud::Dataprovider::Variable

While thinking what would be the most simple data provider to test resource type development, I came up with the idea of a pseudo resource that wouldn’t do anything but allow setting the state at resource creation/update and then return it’s value with GetAtt -call.

Demo & Implementation

AWSTemplateFormatVersion: 2010-09-09
Description: Nordcloud-Dataprovider-Variable

Parameters:
  
  MyValue:
    Description: MyVar Content 
    Type: String
    Default: HelloWorld

Resources:

  MyVar:
    Type: Nordcloud::Dataprovider::Variable
    Metadata:
      Content: !Sub "This could be non-trivial combination of multiple inputs but now it is just a simple Ref to ${MyValue}"

Outputs:
  
  Output:
    Description: Content of MyVar
    Value: !GetAtt MyVar.Content

Code for Nordcloud::Dataprovider::Variable is available at GitHub.

In summary the workflow goes like this

  • Install CFN CLI and dependencies
  • Initialize a new project with cfn init
  • Write the resource schema and generate the handler skeletons with cfn generate
  • Implement the logic in handler functions
  • Validate the resouce type with cfn validate
  • Deploy the new version of resource type and set it as default with cfn submit --set-default
  • Deploy a template using the resource type
  • (cleanup old versions of resource type)

Nordcloud::Dataprovider::Variable doesn’t have much code in handlers. All other handlers are really just a dummy functions returning success, except the read_handler that will return Content value of resource Metadata. This way, I was able to avoid implementing any resource to store variable state but could get Cloudformation take care of it for me.

What Next?

There is a long list of ideas for more serious and useful data providers e.g. finding an AMI, VPC or subnet ID based on given attributes, or mapping between HostedZoneName and HostedZoneId. Or maybe, instead of creating separate types for each use-case, have a generic data provider resource to get attributes of any AWS resource needed.

Resources

  • Cloudformation Provider Development Toolkit and repos for Java/Python/Go -plugins. While cfn cli is usable at it’s current state, GitHub repos can provide valuable support. Java seems to be the most mature language for resource type development and popular for AWS resources too. It is more difficult to find good examples written in Python or Go.
  • Build your first AWS Cloudformation resource provider re:Invent 2020 session describes the details how resource types works. I found this helpful in understanding the callback mechanism that will be necessary for any non-trivial (de)provisioning processes.