Cloud computing costs are difficult to grasp. Most of us have just given up on trying to understand clearly where all the cost figures on your bill come from. You take them for granted and raise an alarm only when it goes totally out of control. Being reactive instead of proactive is what most of us do when it comes to Cloud costs. Sounds familiar? Ever received an unexpected huge invoice? Then this article is for you, because Google Cloud Platform (GCP) allows you to be in full control of your cost through different tools. I’ll explain you how you can become a Cloud cost-control guru on GCP and start your cloud cost optimisation.
Control the things that you can; cost on GCP is one of them
This post is co-authored by our amazing intern Yannis Pagès.
At Fourcast we are consistently using the following tools and techniques to help customers control their Cloud costs in an efficient and accessible manner.
Cost Control Dashboards
Finance controllers love dashboards, so make them happy. We are not going to reinvent the wheel and GCP provides a full tutorial to get a basic version of cost control dashboards. But this is how I recommend to proceed to get your cost control dashboards up and running.
First of all start by exporting your billing data to BigQuery. This will export every single cost transaction to a table in BigQuery at the most granular level you can imagine. From there you can run SQL queries and get quick answers to any questions you have regarding your billing (ie “how much do my VMs in Europe consume?”, “How much do I pay for all the reserved IPs I have?”, etc.). If you are not familiar with SQL, this link will get you started with pre-made billing queries.
Once your billing data is in BigQuery, next step is to create dashboards in Data Studio. You can find convenient template dashboards here.
You can easily get more advanced/customised dashboards by playing around with Data Studio. Having all the data in BigQuery, it’s just a matter of plotting it the way you want.
For instance, if you have some applications running on Kubernetes, you can activate the “usage metering” on your cluster (follow the instructions on this link).This will export detailed usage profile data to a table in BigQuery. By joining data from this table to the original billing data table, you will get the data you need to grasp how your money is spent in your different clusters. And again, Data Studio at the rescue to plot it.
At Fourcast, we have developed some advanced dashboards that enable to fully understand how the costs are split within your cluster. Reach out to us if you want to know more about it.
Billing alerts through Slack notifications
Slack has gained major popularity in the dev community and is certainly a tab that is constantly open on your computer. If you want to convey information to someone, you can be sure that Slack is the way to go. Therefore, why not keeping your team posted on their GCP cost with automated notifications in a Slack channel when they reach some thresholds about a maximum budget?
It’s easy to do exactly that, and I have made things convenient for you by wrapping all what you need in a deployment manager config file. First a quick architecture of the whole setup:
Looking for the github? I got you covered. It’s here.
- Start by creating a Slack app for your workspace, activate incoming webhook and copy the Bot User OAuth Access Token (you will need to replace it in the values.yaml; BOTTOKEN value)
- Replace the values in the “values.yaml” file with your own Slack and project values. Only the first four values are mandatory to change; the rest can be left as default.
- Install the Slack app you just created in your Slack workspace and invite it to the channel where it will send the alerts.
- Install ytt templating library (you won’t regret it, it’s an amazing templating tool)
- Run the following command at the root. It will inject the values from “values.yaml” into the different configuration files and create a new folder (“output-dir”) with the files needed for the deployment.
- Activate the deployment manager API
- Run the following command at the root (don’t forget to replace your project ID)
This will deploy respectively a Pub/Sub topic, a Cloud Function and a table in BigQuery (the resources in the blue rectangle above). As you don’t want to receive notifications every 30 minutes (frequency at which Pub/Sub receives data from billing), the Cloud Function will check with BigQuery if it has already sent a notification for the current billing alert; if yes, no notification is sent to Slack.
Last but not least, you need to create your billing alerts through the billing UI and tick the box to send this data to the Pub/Sub topic created with deployment manager (topic billing-alerts).This step will feed your Pub/Sub topic with the relevant billing data and is the first step in the architecture above.
Slack slash commands
Now that we have dashboards and automatic alerting messages, let’s give anyone the possibility to interact with your billing via Slack; this is done through slash commands.
Slash Commands are initiated from the message box in Slack, but they aren’t messages. A submitted Slash Command will cause a payload of data to be sent from Slack to an app, allowing the app to respond in whatever way it wants. – Official Slack documentation
Slash Commands are easy to set up from an new/existing application into Slack API : you just have to precise the command name. Then, to get Slack linked with your GCP project, create a Cloud Function with an HTTP Trigger set up with the “Request URL” value from your slash command. Doing so will trigger your Cloud Function every time you will run the slash command from any channel of the Slack workspace(s) where the Slack app is installed.
What’s left to do afterwards? Well, I guess it’s time for coding! Indeed, to get Slack commands to work, these are the only two tasks to do. If you’re working with BigQuery, the coding process can be even faster as the function “getBigQuery” you can find in our repo is all you need to query your BigQuery instance from your Cloud Function.
Slack commands for +3 seconds computing time
However, be aware that Slack commands are timing out after running for 3s. If your query needs less time to compute, then stick to the structure of our “current-billing” function.
Nevertheless, for bigger computing time, you may want to look at our “delta-billing” function. In that case, you will need to send a package before processing your query to not have your command cancelled and then send another HTTP message manually at the end of your function. To know everything you need about delayed messages, please check this link at the Slack official documentation.
Deploying functions
OK, time to deploy two functions we have built for you and that you can use as Slack slash command. To deploy the first one, run the following gcloud command:
Once deployed, you can retrieve the URL of this function by clicking on the function itself. Keep this URL close and go in the Slack app you created. Click on the “slash command” section and create a new one. The most important parameter is the Request URL. Paste the one from your function. Specify as well the command as “/currentbilling” and save the command. Your users can now ask for the total amount spent on GCP for this month by writing “/currentbilling”.
We have built another function that gets you the delta billing between two dates. It’s useful for instance to check each day how your billing has increased from the day before, and to pinpoint quickly an anomaly that you can then deeper investigate. To deploy this function, run the following gcloud command:
Then repeat the steps you did for the previous slash command. Copy the URL of the new function, create a new slash command, etc. This slash command expects two date arguments after the command. For instance, if you named your command “/deltabilling” then to get the difference between the 05/07/2019 and 06/07/2019, you will have to write:
“/deltabilling 2019-07-05 2019-07-06”
You have now transformed your Slack into a cost monitoring center for your GCP. Feel free to create more functions and make it even more powerful.
If you have read until here and implemented these three solutions, you are now mastering your Cloud costs in a proactive way. You have dynamic dashboards to monitor and get insights about your Google Cloud Platform costs, coupled with dynamic notifications and slash commands in Slack to warn you about budget alerts. By mastering your costs, you will also find new ways to cut them in efficient ways.
If you need any assistance in either building customised dashboards or Cloud Functions deployment, feel free to reach me at charles.verleyen@fourcast.io. I will be glad to help you.
Want to learn more about Google Cloud Platform?