How We Saved Over $1,000 By Building CloudForecast With Serverless And AWS Lambda

In a previous blog post, we described our journey to launch CloudForecast and some of the questions every startup face. In this post, we will answer the question “What’s the right way to build this product with our resources?”.

The “$100 Startup” concept

I’ve been obsessed with a book called “$100 Startup” by Chris Guillebeau for over a year now. In his book, Chris advises his readers to get their product out fast while being cost-effective to limit the damage if your idea fails (definitely possible, unfortunately!). We try to apply this advice on every side of our business including our technical decisions.

CloudForecast’s Goals

With the $100 startup concept in mind, we started by listing our requirements that were important to us while building CloudForecast:

• Ship it out fast: We were excited to build CloudForecast to help companies save money on AWS. We wanted to put it in front of our clients ASAP to keep our excitement going. Spending time debugging config, deploy process, etc. will not make the end product better. We were looking for an out of the box solution that we can grow with and allowed us to focus on our MVP the right away.
• Cost effective: As an early startup, we wanted to avoid fixed costs by building a system that will scale seamlessly based on our client base.

AWS Lambda + Serverless = Easy + Focus + Cost Effective

With our next steps, we brainstormed on how can we achieve our goals and requirements. We discussed managing our own instances, using containers and half a dozen other ideas but we wanted something simple so we decided to use a Serverless solution (or FaaS). While there are few downsides (e.g cold starts, …), we believe going Serverless better suited our use case (zero administration, pay-per-execution with no idle cost and auto-scaling).

For the dev setup and deploy process, we decided to use Serverless
Framework with AWS Lambda for the following reasons:

• We can focus on writing our product and let Serverless manage the rest
  (permission, event management)
• Easy to configure and easy to deploy
• Serverless supports multiple platforms (GCP, AWS, … ) which could prevent
  headaches in the future. We picked AWS Lambdas to start with since our clients
  data will be stored in S3 bucket. It was a logical decision in order to reduce the
  reduce the network cost
• Great documentation and community (Github, Gitter, Slack)

Our original requirements were fairly simple: We needed 4 AWS lambda functions
each on their own cron job, and each lambda would need to talk to various AWS
products (RDS, DynamoDb, SQS et al.). All that with an easy way to manage
multiple environments (dev vs prod) easily with an easy/effective way to manage
resource permissions.

Here is how we do it:

• We used 4 functions with external JSON files variables based on the environment:

• Used the
  AWS Lambda functions code snippet to configure all the permissions:

With these two code snippets, we were able to set up most of our architecture. 4 functions that will interact with each other through SNS. Two functions will run check on a cron schedule (via the ‘schedule.rate’ params) to check a file need to be reprocessed and trigger another function via an SNS. This configuration will be able to scale effortlessly while keeping our cost under control. We are able to fully silo-ed our environments using iamRoleStatements to configure the permissions. We used an external variable to manage environment variables from a simple YAML file.

We considered running a couple of small instances to do a job which would cost us at least $1k for the year. Meanwhile, the AWS Lambda cost is meaningless since we are only running tens of thousands of functions a day.

What’s Next?

Using Serverless helped us get CloudForecast off the ground quickly. We saved our first clients thousands of dollars by giving them visibility into their AWS spend.

We are in the process of integrating AWS Athena into CloudForecast for our next project. Integrating with AWS Athena will enable new features that will allow end users to dive deeper into their AWS spend. Every company has unique spending patterns, and this will allow them to fully understand where their money is going. Check out our new blog post on how to save money on AWS by using Parquet on AWS Athena.

If you have any questions related to this post or what we do at CloudForecast, feel free to reach out to me at [email protected]. We would love to hear from you!

-Francois & the CloudForecast Team

Haven’t tried our features? Sign up today and get started with a risk-free 30 day free trial. No credit card required.

Special Thanks to Kyle Galbraith for reviewing this draft. Check out his weekly Newsletter, Learn By Doing. Learn by Doing is a free weekly curated cloud, blockchain, and coding newsletter delivered to your inbox every week.