Update 3/23/2018 The AWS CodePipeline team has updated their documentation to indicate that it is, in fact, possible to use IAM resource-level permissions on approvals. I’ve updated the post below to explain how this works, and the remaining gotchas.
You might be alive today thanks to the ”two-person rule” - it’s the US military policy that requires two keys to launch a nuclear missile, so one crazy officer can’t start World War III.
Maybe your business’s software isn’t quite as dangerous as a nuclear warhead, but you’d still prefer it didn’t blow up in your face. For that reason, you might choose to integrate some form of the two-person rule as part of your CI/CD pipeline: requiring multiple people to sign off manually on a code change before it enters your production environment.
I’ve been working recently with CodePipeline, an AWS service that automates code builds and deployments in the cloud. When I saw that CodePipeline supports manual approval actions, I thought: “How hard could it be to implement the two-person rule as part of a pipeline stage?”
Implementing Multiple Approvers in AWS CodePipeline
Since we’re using IAM authentication for users in our AWS environment, a two-person approval solution should meet the following criteria:
CodePipeline must record two or more manual approvals before successfully exiting a pipeline stage
Each approval must be associated with a valid IAM user or role
The approvers must be different: the same IAM principal should not be able to provide more than one approval for the execution of the same pipeline stage
IAM-Based Approvals
The most logical way to set this up would be to place two approval actions in parallel at the end of the pipeline stage, looking something like this:
You could then create two IAM roles or groups and give each group permissions to approve only one of the two approval actions. Now you can just divide your administrative users among the two groups and you’re good to go.
Unfortunately, AWS CodePipeline does not currently support resource-level permissions for the “PutApprovalResult” IAM action. CodePipeline actually does support this! The trick is that the IAM visual policy editor doesn’t know that, and will show you a warning saying that resource-level permissions are not supported. To set up resource-level permissions, you’ll need to define two IAM policies in the JSON editor as follows:
You can attach these policies to two different IAM groups or roles, and divide your IAM users between those groups. As long as an individual user belongs to a maximum of one approval group, and you keep a consistent naming scheme (in this case, ApprovalA and ApprovalB) for your pipeline approval actions, then any user in the A group who tries to approve a B action, or vice versa, should see this error message:
Boom! You’ve got the two-person rule working, and all it took was a few lines of JSON. Not bad!
Lambda Approval Checks
The only issue with the above approach is that it’s dependent on your IAM admins making sure that no single IAM user has access to approve both actions in a pipeline stage, and that your action names are consistent across all pipelines. Wouldn’t it be nice if there was some way we could programmatically enforce different approvers in our pipeline itself?
Essentially, we just want to be sure that two different IAM entities performed our two approval actions, and throw an error if they don’t match. This is a relatively easy check using the GetPipelineStage API call as shown in the example Python Lambda function below:
from __future__ import print_function
import boto3
import traceback
code_pipeline = boto3.client('codepipeline')
def put_job_success(job, message):
code_pipeline.put_job_success_result(jobId=job)
def put_job_failure(job, message):
code_pipeline.put_job_failure_result(jobId=job, failureDetails={'message': message, 'type': 'JobFailed'})
def lambda_handler(event, context):
try:
job_id = event['CodePipeline.job']['id']
state = code_pipeline.get_pipeline_state(name='YOUR_PIPELINE_NAME_HERE')
stage = state['stageStates'][1]
first_approver = stage['actionStates'][0]['latestExecution']['lastUpdatedBy']
second_approver = stage['actionStates'][1]['latestExecution']['lastUpdatedBy']
if first_approver == second_approver:
put_job_failure(job_id, "ERROR: This pipeline does not permit the same approver ({}) to provide two approvals!".format(first_approver))
else:
put_job_success(job_id, "Two different approvers signed off on this change: {} , {}".format(first_approver, second_approver))
except Exception as e:
# If any other exceptions which we didn't expect are raised
# then fail the job and log the exception message.
traceback.print_exc()
put_job_failure(job_id, 'Function exception: ' + str(e))
return
NOTE
The example implementation above would only be guaranteed to enforce unique approvals between IAM users. It might require some administrative effort on your part to ensure that roles with permissions to approve CodePipeline executions are associated with distinct sets of users. You would also have to think about sessions - you would not want a user to assume the same role twice using different session names and have your code recognize that as two different roles. These considerations are left as an exercise for the reader. :)
Now we can chain this function into our pipeline. As long as the approvers are different, the pipeline will succeed as shown below:
If the same user tries to approve both manual actions, the pipeline execution will fail with an error like this:
Further Limitations
We now have a solution that fulfills our primary goal: stopping a CodePipeline execution unless it gets manual approvals from two distinct identities. However, it’s still not an ideal solution. Failing the whole pipeline execution just because somebody tried to approve both actions is a bit drastic - after all, especially if you’re spinning up a bunch of CodeBuild jobs, a CodePipeline run can take quite a bit of time. It would be much better if we could just deny the second approval attempt and keep that approval action open for review by another authorized person, essentially pausing the execution instead of outright failing it.
CodePipeline does support an API call to retry a particular stage, but as of now it only retries failed actions in that stage. There’s no way to arbitrarily mark a “successful” action as failed from a later point in the pipeline execution, or vice versa - which is probably a good thing - but it does mean that we can’t use our Lambda function to reach back and mark that second approval action as failed so we can retry it.
But now we’re just nitpicking. Ultimately, using a combination of IAM-based approvals and Lambda sanity checks should give you a robust and secure implementation of the two-person rule. Your production environment can breathe a sigh of relief - doomsday has been averted!
