Maintain AWS Tags When You Fall Behind – Part 3

After adopting an AWS tagging strategy, you face two new challenges: finding improperly tagged AWS resources and enforcing your tagging strategy going forward. In the previous installment of this guide, you saw how to enforce your AWS tagging strategy when creating new resources with Terraform or CloudFormation. In this part, you’ll see how to find improperly tagged AWS resources.

Tagging your resources in AWS will help prevent misuse and maintain your infrastructure. When combined with a tool like CloudForecast, tagging your resources can help you predict and control your monthly spending, so it’s usually worthwhile to invest in tag maintenance.

Unfortunately, it’s easy for AWS tags to get out of date. If you’re adopting a new tagging strategy, you likely have a lot of catching up to do, but even if you’ve been following one for years, things can get stale. Your team may want to change the name of a tag or update the rules surrounding certain tags. Developers may leave without handing off resources they managed, or someone might forget to change an outdated tag on an AWS resource. Even the best AWS tagging policy will require ongoing maintenance, especially as the number of resources you manage grows.

To help maintain your AWS tags, you should regularly audit them to make sure they’re still accurate. At a small company, it might be possible to do these audits manually, but you’ll need to use tools to help you automate the process as you grow. This article will show you six tools you can use to help find and fix outdated AWS tags. You’ll see some of the use cases for each so you can decide which ones are best for your organization. I’ll do a deep dive into the first two because they’re the most commonly used today, but it’s a good idea to have a few options to choose from.

1. Cloud Custodian
2. AWS Config Rules
3. Retro Tag
4. Gold Fig
5. AWS OrganizationTag Policies
6. CloudForecast Tagging Compliance Report

Looking for practical help with AWS Tags? Learn how our Tagging Compliance feature help users implement best practices when it comes to AWS tags

Cloud Custodian

Cloud Custodian is an open-source collection of scripts that help developers manage their public cloud accounts. For this article, I’ll limit my focus to Cloud Custodian’s tag maintenance features in AWS, but it supports a variety of use cases and most of the major cloud hosting providers.

One use case for Cloud Custodian is to find AWS resources that don’t comply with your tagging policy. Let’s say you have an application consisting of an elastic load balancer and three EC2 instances like this:

And you want all your EC2 instances to:

• Have an env tag with a value.
• Have a contact tag with either j-mark or l-duke as the value.
• Have a service tag with either cart or search as the value.

Cloud Custodian’s policies are stored in YAML files so there’s no state to maintain outside of the policy itself. To create a policy that will tell you if any of your EC2 instances aren’t in compliance with the above rules, create a new policy file called policy.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

policies:

- name: ec2-tag-compliance-report
  resource: ec2
  comment: Report on ec2 instances without required tags
  filters:
    - or:
      - "tag:env": empty
      - type: value
        key: "tag:contact"
        op: ni 
        value: ["j-mark", "l-duke"]
      - type: value
        key: "tag:service"
        op: ni 
        value: ["cart", "search"]

You can run Cloud Custodian from your local machine or set up a cron job to run it from a server on a recurring basis. Once installed, run the above Cloud Custodian policy from your command line:

1

custodian run -s ./ policy.yml

Cloud Custodian will output a “count” of the number of EC2 instances that are missing one of your required tags and generate a report in your output folder (./ in this case) that includes more details about the resources so you can remedy their tags.

Cloud Custodian can also perform actions to remediate or alert you to issues automatically. For example, you may want to call a webhook that triggers a notification when an EC2 instance doesn’t adhere to your tagging policy.

To add a webhook action, replace your policy.yml file with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

policies:

- name: ec2-tag-compliance-webhook
  resource: ec2
  comment: Trigger a webhook when ec2 instances are incorrectly tagged
  filters:
    - or:
      - "tag:env": empty
      - type: value
        key: "tag:contact"
        op: ni 
        value: ["j-mark", "l-duke"]
      - type: value
        key: "tag:service"
        op: ni 
        value: ["cart", "search"]
  actions:
     - type: webhook
       url: https://example.com/hooks?id=1

Now when you run Cloud Custodian, it will call https://example.com/hooks?id=1 if any EC2 instances fail to adhere to your policy. There are many examples in their docs for using filters and actions to perform other tasks. While running Cloud Custodian locally from your command line is a good way to test your configuration, you should ultimately deploy your policies to a server to run them against your production environment on a schedule. You can even deploy Cloud Custodian policies as a periodic Lambda function to help minimize the cost.

AWS Config Rules

Amazon’s official tool for auditing your resources is AWS Config. Like Cloud Custodian, it can be used to check a range of compliance and configuration settings, but I’ll focus on its managed rules for requiring tags.

Using AWS Config, you can specify which resources should have tags and the expected values for each tag. AWS Config allows you to run remediation steps when a violation is found so that your team can quickly fix tagging mistakes, but it doesn’t prevent you from creating resources with improper tags. For that, you’ll have to go back to the previous installment of this guide.

To reproduce the Cloud Custodian policy above, go to AWS Config > Rules > Add rule. Enter “required-tags” in the search bar and select the required tags rule.

Name the rule, add a description, and select the resources you want this rule to apply to. In this case, enter EC2: Instance only if you just want to make sure your EC2 instances are correctly tagged.

Next, enter the tags and allowed values that you want to enforce. To replicate the Cloud Custodian policy above, set:

• tag1Key = env with a blank value
• tag2Key = contact and tag2Value = j-mark,l-duke
• tag3Key = service and tag3Value = cart,search

Skip the remediation details for now and click “Save.” You’ll be taken back to the list of AWS Config Rules. The initial check will take a few minutes to run, so wait a bit then refresh the page. AWS Config will show you a list of which resources are in compliance and which are not. Be aware that AWS Config can take up to 6 hours to reindex your tags after they’ve been changed on your resources.

If you want to call a webhook like you did for Cloud Custodian above, you will need to create an SNS topic that calls a webhook. After creating the topic, edit the rule and select AWS-PublishSNSNotification.

Add an IAM Role ARN with SNS access to the AutomationAssumeRole field, add a message, and paste the TopicArn from your SNS topic. This will allow AWS Config to trigger the SNS topic references when the config rule fails.

Click “Save.” Now, when new EC2 instances are found that fail this compliance check, the SNS topic will call your webhook. This isn’t the only remediation step available, so read more about your remediation options in the AWS Config docs.

While the AWS Config rules aren’t as powerful as Cloud Custodian’s, there are some advantages. It’s built into AWS and works with CloudFormation. This means there are fewer extra files to manage, and AWS Config provides a visual configuration timeline that will help you track down improper tagging practices.

That said, there are other problems with it like its lack of support for some resources and cost. If you’re deciding between AWS Config and Cloud Custodian, you might want to do a deeper dive into each tool to determine which will work best for your team.

Retro Tag

Unlike the previous two tools, Retro Tag wasn’t built for finding untagged resources directly, but instead, it focuses on helping you track down the creator of every resource in your AWS account. This is an useful step in auditing your tags because the person who created the resource usually knows the most about it. Once you find the creator, you can ask them how the resource is being used (if at all) and put them in charge of fixing the tags.

Retro Tag is open-source and built on the Auto Tag engine by GorillaStack. It works by using AWS Athena to export a CSV of your relevant CloudTrail events. The Retro Tag script uses this CSV to tag any resources that still exist and don’t yet have an AutoTag_Creator or AutoTag_CreateTime tag.

If you’re using Auto Tag to add creator tags to all new resources, Retro Tag is a great way to catch up on resources that were created before Auto Tag was installed. That said, you don’t have to run both of them together. You may just want to add creator tags to old resources so you can assign responsibility for updating tags to the creator rather than doing it yourself.

Gold Fig

While Cloud Custodian and AWS Config audit your resources on the fly, Gold Fig takes a different approach to the problem. It loads all your AWS resource data into a Postgres database and gives you a command-line interface to run queries against it. Gold Fig is also open-source, so you can deploy it to your local machine or a server that automatically syncs the data regularly.

Gold Fig doesn’t offer remediation actions like Cloud Custodian, but developers familiar with SQL might find the query syntax more natural than learning another flavor of YAML. Because loading your data from AWS takes several minutes, it also isn’t ideal for tracking real-time updates to your resources. But, if you’d like to perform periodic audits on your resources using complex logic that’s tricky to build in Cloud Custodian or AWS Config, Gold Fig might be worth a try.

AWS OrganizationTag Policies

Another option if you’re using AWS Organizations is to create tag policies. Tag policies are similar to AWS Config rules, but because they work at an organization-level, you can use them to audit your tagging strategy across multiple AWS accounts. While many AWS resources support organization tag policies, not all do, so be sure to check the list before you go down this path.

CloudForecast Tagging Compliance Report

Finally, if you’re embarking on a tagging improvement project, you might want to start off with a free tagging compliance report from our team at CloudForecast. We can help you find untagged AWS resources and ensure that everything in your AWS account is accurately tagged. Once you’ve got your free report, CloudForecast sends you a daily AWS cost report and savings plan so that you can spot opportunities to save money on your AWS bill every month.

This compliance report will save you a lot of time as looking for gaps in your tagged resources can be pretty mind-numbing. We’ll help you quickly discover untagged resources and sort them by cost so you can fix the low-hanging fruit and increase compliance.

Conclusion

While managing all your resources in Terraform or CloudFormation, will help you create resources with the right tags in the future, you’re probably not starting your AWS account with a clean slate. Maybe you started out creating resources manually via the CLI and only recently adopted infrastructure automation. Maybe some teams are using CloudFormation, and others prefer the GUI. At some point, you’ll need to find and fix your improperly tagged resources.

The tools above should help, but if you’re still lost, feel free to reach out to our CTO, Francois. We’d love to help you set up your organization’s tags for success and are happy to offer a free tagging compliance report as well.

Looking for practical help with AWS Tags? Learn how our Tagging Compliance feature help users implement best practices when it comes to AWS tags