- Commit Signing with Git at Enterprise Scale
- What Does Git Need for Commit Signing at an “Enterprise” Scale?
- How were the new features implemented?
Commit Signing with Git at Enterprise Scale
Git is one of the most ubiquitous version control systems used today, seeing extensive usage in projects both around the world and within Arista. Everyday numerous Arista employees, located around the world, make commits to the codebase to fix bugs, add features, and save works in progress. The same scenario plays out with many other people, both when working for private enterprises, government institutions, and open source projects. The following paper discusses changes made to alleviate a fundamental security problem with Git, and version control systems in general. It is assumed that readers of this paper will have some knowledge of computer science, version control systems (VCS), and limited knowledge of cryptography.
During an internal security assessment one fundamental issue was noted with Git; It is very hard to prove that a commit actually came from the author named in the commit. This is problematic since that means that a malicious adversary could, upon gaining access to the codebase, add code under valid employees names. As time goes on from when the code is added, this becomes harder to ferret out. To explain why, consider the mathematics of the situation. Assume a company produces 100 commits per business day. Now assume that the adversary is caught after one week, during a review of suspicious logins. There are now 500 commits to go through, since the adversary could have attached any author name to the commits they added. If the number of commits or time before being caught increases, the number of commits to examine scales up. Examining a large number of commits can become an insurmountable problem when looking for small, suspicious changes. To be fair to Git, this is an issue that many other version control systems have. Most version control mechanisms predate a strong understanding of this issue, which is a type of threat known as “supply chain attack”.
Traditional security mechanisms around version control systems involve a secure perimeter combined with a gatekeeper. The perimeter can be a firewall, physical separation, or some other mechanism to prevent write access to the VCS. Any request to write to the VCS must go through a gatekeeper that validates the commit and the committers identity. Trust in such a system relies on the gatekeeper being trusted over time. If there is found to be an issue in the gatekeepers ability to correctly validate users, it calls into question the validity of the commits. In addition, administrators of the VCS will often have direct write access, also bypassing the gatekeeper. If an administrator’s credentials are stolen, arbitrary code can be added to the VCS.
Git also has a feature that can help resolve this problem. Known as commit signing ( https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work ) this feature allows for individual commits to have a cryptographic signature placed on them, stating “I, the author, created this commit with the following contents”. A signed commit gains two properties: The author is associated with the commit and the contents of the commit cannot be altered. Should the commit be altered post signing, it will fail to pass a check, known as “validation”, when the signature is examined.
This feature solves the threat of a Git repository being written to by a malicious person. In order to write to the repository an employees code signing key must be obtained. Once a malicious commit is identified, only that employee’s work needs to be examined. There is a caveat however; only users who sign commits can have their commits validated. This means that for this to be an effective protection, all commits must be signed. The process of forcing all commits to be signed in a repository is what creates the problem of commit signing at an Enterprise scale.
What Does Git Need for Commit Signing at an “Enterprise” Scale?
To enable commit signing to scale up across an entire enterprise, better support around Git is needed. Git itself works wonderfully, but to enable it to work additional infrastructure needs to be added to manage keys, commits, and auditing the signatures on commits in the repositories.
What is commit signing like with Git today?
Git currently supports commit signing for individual commits. If a commit is altered for any reason; rebasing on other commits, underlying file store fails, malicious additions, then the signature will fail to be validated. The original author will need to re-sign their commit in order for it to be validated once again. These are desirable properties for proper commit signing in a VCS. However, knowing how to validate someone else’s signed commit is where issues begin to arise.
The cryptographic methods used to sign the commits are based on PGP ( https://en.wikipedia.org/wiki/Pretty_Good_Privacy ) which means that a “web of trust” model is used for trusting keys. “Trust” is defined here to mean that a public key can be tied to someone’s identity. A web of trust means that there is no central authority that determines which keys are valid and tied to the users identity. Instead a user, Ingo, can choose to trust others, call them Eva and Axel. Ingo will verify the public keys of Eva and Axel and thereafter be able to validate that a commit coming from Eva truly came from her and was not altered. Ingo can also choose to trust people who Eva or Axel trust. If Axel trusts Manuel, Eva can add a signature to Manuel’s public key stating her trust that the public key Manuel showed Eva belongs to Manuel. Manuel can then show a signature and his public key to Ingo. Since Ingo trusts Eva, and Eva trusts Manuel, Ingo can validate the signature and know that the signed data came from Manuel. This means that to become an accepted member of an organization, a user needs to find someone willing to sign their key. As the size of an organization scales up, this creates issues with understanding: Who is allowed to sign keys? Who does a particular user trust? How does that trust change over time? If an adversary can convince any trusted person to sign the adversaries key, the adversary can infiltrate the organization.
Keys are also required to be generated by the user in the PGP model. Since PGP is oriented around not having a central authority, this is sensible in the design case. However this has the problem that the keys themselves may not be generated in a secure manner. One common problem is that the system may be in a state of “low entropy” where random numbers used to generate keys may not meet a minimum standard. The key generation algorithm itself may also have problems. These sort of problems are hard to determine from examining the keys, but may result in issues later, especially when needing to investigate generated keys. Another issue that arises is that the keys are then managed by the user. If the user stores their keys on a laptop, and that laptop is stolen or breaks, they will need to generate new keys and start over in their process of getting that key trusted by others within the organization.
What Does Enterprise Scale Commit Signing Look Like?
In order for commit signing to be effective with Git, every commit must be signed by a trusted key. Every key must be managed by a central authority in order to be able to provide services such as key generation, revocation, and rotation. All previous items must be auditable at future points in time and minimize the size of any security perimeters.
In order to accomplish this the following support tooling was built around Git:
- Central Key Management – Hashicorp’s Vault (https://www.vaultproject.io/) is a well known server used for managing secrets. While Vault does not provide an interface for managing PGP keys by default, plugins can be used to extend Vaults functionality. Vault is then used to generate a master key for each user along with a commit signing “subkey”. Subkeys are a feature of PGP discussed later on in this paper which allow for Vault to manage the keys while still allowing each person their own key.
- Only allow signed commits – Arista uses a combination of Gerrit (https://www.gerritcodereview.com/) and Jenkins (https://www.jenkins.io/) to manage internal repositories. Gerrit manages the Git repositories and shows code reviews. Jenkins performs automated testing against each review. It is required that each commit merging into a repository pass a code review with testing from Jenkins. By adding a test in Jenkins to check the signature on each commit, it is possible to enforce every commit going into the repository is signed.
- Audit signed commits – In order to make sure that all commits going into the repository are signed, and nothing was added or altered, an auditing mechanism is needed. This mechanism periodically goes over all merged commits in the repository and validates that each one was signed with a key that was valid at the time of signing.
How were the new features implemented?
Central Key Management
The goal of centralized key management was that the organization should be responsible for the management of keys rather than the developers. This means that the organization should issue keys, rotate keys, and revoke keys as needed.
Implementing the ability for the central organization to be able to control keys required using a PGP feature known as “subkeys”. Subkeys are PGP keys which are bound to a master key and can have limited permissions decided at their time of creation. For example, a subkey could be created which would only be able to sign commits. The master key is able to decide when to create subkeys as well as revoke them. Each developer is issued their corresponding subkey with permission for commit signing and can view the public portion of other master keys and commit signing public keys as they need to. With this setup the central organization is able to alter the validity of the issued subkey as needed. For example, if the developer has their laptop stolen the developers master key issues a revocation notice for the developers current code signing subkey and generates a new subkey. Since all keys are generated by the central organization there are no concerns over keys being generated with poor entropy due to unknown circumstances behind their generation.
Vault is written in Go (https://golang.org/) and plugins for Vault are also written in Go. There is also already a plugin for Vault which manages GPG keys (https://github.com/LeSuisse/vault-gpg-plugin/). We took the approach of extending that plugin to add the additional features we needed, mainly revolving around support for subkeys. Those changes can be viewed here: https://github.com/LeSuisse/vault-gpg-plugin/pull/18 . Due to requiring changes in the upstream golang pgp library those changes have not been merged as of the time of this writing.
As hinted above, the Go libraries needed to be extended as well to support the features for central key management. Those changes also involved adding subkey support to the Go library. Those changes are available at https://go-review.googlesource.com/c/crypto/+/161817/ and have also not been merged as of the time of this writing.
Validation of Signatures
Every commit needs to be signed in order for commit signing to be effective. At Arista the Gerrit tool is used to manage code reviews going into git repositories. All commits merging to the git repository must pass a gerrit code review, which includes a human review and automated tests.
To make sure that the commit is signed before merging, an additional test is added which does the following:
- Look at the commit metadata, including author email and commit signing key fingerprint.
- Use the commit signing key fingerprint to obtain the full public key from the central key database.
- Examine the public key to ensure that the author email matches the signing keys owner
- Validate that the signature over the commit is valid.
Any failure of the steps will mean that the commit will not be considered valid to merge.
Auditability of Signed Commits
Central Key Management and Validation of Signatures set up a gating mechanism for ensuring that all commits going into a repository are signed. However, if Gerrit or the git repository are compromised, additional commits could be added or previous commits tampered with. Git repositories at Arista use the “rebase” style to merge new commits in. This leads to a clean commit history, but loses signature information in most merges, since rebases redo a (signed) commit on top of the current head commit. Redoing the change will create a new commit, invalidating the previous signature.
This issue is solved by making use of Gerrit’s detailed information and APIs. Every commit Gerrit manages has a change-id associated with it. You can see an example at https://go-review.googlesource.com/c/crypto/+/161817/. Then change-id is itself a link, which leads back to the review. The review contains the signed commit. All the needed information to verify the audit is now available.
The process used to audit the codebase is as follows:
- Start with a specific commit in the repository.
- Using the change-id in the commit, find the gerrit change associated with the commit.
- Pull the signed commit from the gerrit change.
- Perform the validation steps discussed in validation of signatures.
- One small change from the prior validation algorithm needed is to ensure that the keys used were valid at the time the commit was made. Keys can be rotated over time and Vault acts as a store of all keys ever generated and notes when they were generated.
- Create a copy of the repository which is one commit before the commit being examined.
- Replay the commit being examined on top of the repository copy.
- Ensure that the repository with the replayed commit is identical to the original repository with the commit.
- Repeat the process with the next commit in the repository.
If at any point this fails an email is sent out the security team to examine further. By performing this audit it can be validated that there were no commits made directly to the git repository that skipped the validation of signatures step. Any commit which skipped that step would either:
- Not have a valid change-id
- Not have a valid signature.
In addition, if commits were changed at any point, even if the gerrit review system was compromised and reflected the altered commit, the signature check would fail.
This project was successfully deployed and is currently used at Arista as part of our strategy on preventing supply chain attacks. Throughout the project we faced various issues that we felt would be of interest to share here in a postmortem.
- Lack of interest from Golang upstream: One of the early surprises when we extended the PGP libraries in Golang was the lack of interest in accepting the changes from upstream. At the same time that we working on this the Golang team was interested in putting the pgp crypto libraries into the /x/ sublibraries and finding a new group of maintainers: https://groups.google.com/g/golang-openpgp/c/_P6AmeCmD9w. Eventually we were able to learn that Protonmail (https://protonmail.com/) was doing their own work on extending the pgp library and they were happy to accept our changes. Those changes are now in the process of making their way back to the golang main branch.
- Don’t trust other services: A lot of security issues arise from blindly trusting other services to do the “correct” thing. This proved true during this project as well.
- Disabling the validation job in the review pipeline: During the initial rollout of the validation job in the jenkins review pipeline, reviews that were not signed with a valid signature would sometimes slip through. This was later tracked down to be the result of frustrated engineers disabling the job. Fortunately the audit verification job would catch these later on. After identifying the engineers the best solution was to reach out to them, understand why they got frustrated, and help resolve the problems they were experiencing with the commit signing. These resolutions would normally take the form of enhanced tooling or better explanations around what needs to be done to sign a valid commit. After a few rounds of this, those problems were resolved.
- Perform your merges yourself: Gerrit provides a feature to determine if a rebase is a trivial rebase (no changes to the commit) or a non-trivial rebase (with some changes made to where the commit is merged). This feature is also fairly easy to trick into returning incorrect data if someone has administrative access to Gerrit. This was noted during the creation of the audit verification job and is why the audit job performs all its merges itself to check the results.
- Convincing engineers to use this: Having everyone in an organization sign their commits is a major change. There were questions around key distribution, changing git files, installing gpg, and a disruption to each developers throughput if they did not follow this correctly. The solution that we found to this was to combine a soft rollout with educating developers. Most people, when understanding the threat this resolves, were fine with the changes required to sign their commits. A small set of people needed more personalized discussions to resolve their concerns. By having this be an open discussion and making ourselves reachable we received a lot of great feedback and suggestions from everyone as we worked on rolling this out. Some of the biggest critics of the effort required turned out to come up with some of the best feature suggestions for how to integrate this into their workflows. Repositories also had a gradual rollout process where commit signatures were validated, but not needed, so that developers could get used to signing commits before the go-live. By combining the soft rollout, open discussions, and turning feedback into new features, we were able to have a successful rollout with very minimal disruptions to workflow by the time signatures were required on every commit.