Rogue employees running your cloud infrastructure can skim money off your monthly bill. Here’s what you can do to prevent this fraud and unnecessary cost.
How Cloud Cost-Skimming Fraud Works
Cloud compute services like Microsoft Azure and Amazon Web Services (AWS) allow developers to publish their own virtual machine (VM) templates to their global marketplace so that others can consume them. This lets legitimate software vendors create easy-to-deploy, cloud-ready installations of their software. Customers get easy access to useful software and vendors get paid for their product. Not all marketplace templates or images have an additional charge for using them, but many do.
Premium Virtual Machines
Cloud providers offer a set of base templates or images that are billed based on usage of the underlying hardware and any operating system licensing costs. These generally come in a variety of Windows and Linux flavours. Developers who release their own premium virtual machines can add an additional charge on top of the Azure/AWS standard charge. With a pay-as-you-go, usage-based model, this can be anything from one penny ($0.01) per hour upwards. A percentage of any additional costs billed to consumers is paid to the developer who published the image.
The following table taken from the Microsoft Azure Marketplace documentation explains to developers how much they will get paid. A similar guide is available for the Amazon AWS Marketplace.
How Employees Can Use This to Defraud You
We must trust our employees in order for them to work effectively. In truth, most employees are trustworthy individuals who would never consider defrauding their employer. That doesn’t mean we shouldn’t define boundaries for our people to work within. Nor should we abstain from an appropriate level of due diligence. Setting clear boundaries allows people to work autonomously towards their goals. Technical controls and monitoring can protect us from dishonest actions by the minority. So what is it that we’re watching for?
Instead of using the standard, no additional charge images provided by Azure and AWS, employees can:
- register as developers on the AWS and Azure Marketplaces
- create exact copies of the base images they should be using
- then charge an additional fee for using them. This could be anywhere from $0.01 per hour to something a lot more significant.
If you’re only running a single virtual machine, an additional $7.44 in costs isn’t going to make much difference to your overall spending. If you’re running tens, hundreds, or even thousands of VMs, spending an additional $0.01 per hour per virtual machine will be a substantial but difficult to detect overpayment by you, and a nice pay rise for the fraudster.
How Can This Fraud Go Undetected?
With more and more reliance on automation tools for the creation, management, and destruction of virtual machines, it’s easy for an attacker to change a base image from the standard offering to a machine of their own. When your deployment process is fully automated, the fraudster only needs to make one change to a configuration management tool to cash in on every VM created from that point onwards.
Cloud compute costs are very variable. The natural variance in a flexible usage model will always create peaks and troughs in your monthly bill. With careful execution, the implementation of such a scheme could be eased in slowly to avoid an immediate rise and deliver a gradual rise over several months.
As anyone can register a company, register for the Azure and AWS marketplaces, and start creating custom, premium images, the trail of evidence leading back to a known individual may not be obvious. If you’re already using premium images in other places, one more vendor isn’t going to raise much suspicion, especially if the costs are almost identical to what you were expecting.
How to Prevent Cloud Cost-Skimming
The methods you use will vary from one cloud provider to another. For some, the only option you have is to manually review detailed usage logs during or at the end of a billing period. Thankfully, for Azure AWS we have other options.
Preventing Unauthorised Purchases on the Microsoft Azure Marketplace
Azure Policy is a service in Azure that you use to create, assign, and manage policies. These policies enforce different rules and effects over your resources, so those resources stay compliant with your corporate standards and service level agreements. Azure Policy meets this need by evaluating your resources for non-compliance with assigned policies… For example, you can have a policy to allow only a certain SKU size of virtual machines in your environment. Once this policy is implemented, new and existing resources are evaluated for compliance. With the right type of policy, existing resources can be brought into compliance.
– Microsoft: What is Azure Policy
As Azure administrators, we can use Azure Policy to prevent our employees from launching services by unapproved publishers on the Azure Marketplace. We create a whitelist of known good publishers, people such as Microsoft, RHEL, Canonical, etc., then set a ‘deny’ control for anyone not in that list. As long as our employees don’t have access to change the Azure Policy configuration, Microsoft will not allow anyone to launch their own Marketplace VMs with unexpected charges.
This GitHub repository has several examples of restricting publishers through Azure Policy and this guide does an excellent job of explaining how and why it works.
Preventing Unauthorised Purchases on the Amazon AWS Marketplace
AWS offers a similar solution to Azure. In order to launch premium products, you must first subscribe and agree to the product’s EULA. Identity and Access Management (IAM) policies can be created to prevent all your AWS users from subscribing to new products on the Marketplace, except for a specific group such as enterprise administrators or your purchasing team. With these policies in place, you can create additional groups who can launch specific images which have already been approved by the management team.
As long as you’re AWS and Azure implementations adhere to the principle of least privilege, you can prevent cloud cost-skimming by applying technical controls. For other providers, check their documentation for similar policy-based controls. Failing that, a manual audit of monthly usage statements may be our only option.
