BYTEHERDERScanning your dependencies in CI is good practice. Nobody argues with this. You add a stage, it looks at your lockfile, it tells you about known vulnerabilities, and you find out about the bad one before your customer does.
I believe this. I have set it up for other people. It is the correct thing to do.
It keeps working, too
Add Trivy to the container build and you find the base image has been shipping a three-year-old libxml2. Useful.
Add a dependency scan on the lockfile and you find the transitive pull that nobody chose, four levels down, doing HTTP requests at import time. Very useful.
Add a secret scanner to the pre-receive path and you find the AWS key somebody committed in 2023 and then "removed" in the next commit. Extremely useful, and slightly alarming.
Three for three. The scanners work. They find real things.
The line at the bottom of the stage
So when I am handed a pipeline to look at, the scanning stage is not where I expect to find the problem. I read it the way you read a paragraph you already agree with.
Which is how, on one engagement, I got most of the way down a 400-line GitLab CI file before I noticed this:
container_scanning:
stage: test
script:
- trivy image --severity HIGH,CRITICAL "$IMAGE"
allow_failure: true
allow_failure: true.
I checked the other three. continue-on-error: true on the SAST job. A || true at the end of the dependency check. And the secret scanner, the good one, reporting into a dashboard with no threshold configured at all.
What the pipeline had actually been doing
Here is the number that ended the discussion. I pulled the job history for the previous twelve months. The scanning stages had run 2,310 times. They had reported findings on 2,118 of those runs.
They had blocked a release zero times. Not once. Not a single deploy in a year had been stopped by a security finding, and the pipeline had been designed, from the first commit, so that it could not be.
That team did not have a security gate. They had a security newsletter with excellent open rates and a subscriber base of nobody.
And I want to be fair about why, because it is not negligence. Somebody turned allow_failure on during the rollout, so a noisy new scanner would not block everyone on day one. That is the correct move on day one. The mistake is that day one never ended. There was no date on it, no ticket, no threshold to graduate to. A temporary exception with no expiry is a permanent decision that nobody remembers making.
The part that is not about tooling
The fix is not a better scanner. Trivy was finding the right things. The fix is a decision, and it is a decision the tooling cannot make for you:
Which findings stop a release?
That question is uncomfortable because any honest answer is going to block a release at some point, probably at an inconvenient moment, probably for something you will argue is not exploitable in your context. Which is exactly why allow_failure: true is so restful. It defers the question forever.
Write the answer down. It does not have to be strict. Ours, on that engagement, started deliberately loose:
- Critical severity, fixed version available, in a runtime dependency: blocks.
- Anything else: reports, and appears in a weekly list with an owner's name on it.
- Exceptions: a file in the repo, one line per exception, with an expiry date. CI fails on an expired exception.
That is it. Three rules. The first release it blocked was eight days later, and it was a real one: a HIGH in the JSON parser on the public-facing API, fix available, one version bump away.
SBOMs have the same disease
While we are here. Generating an SBOM is now a checkbox on a lot of procurement forms, so a lot of pipelines generate one, attach it to the artifact, and never look at it again.
An SBOM that nothing reads is an inventory of a warehouse nobody visits. The control is not the document. The control is the thing at admission time that reads the document and refuses the deploy when the answer is wrong. In Kubernetes that is a policy engine checking the attestation before the pod is admitted. Until something is reading it and able to say no, you have a build artifact with a manifest attached, and a slide that says you do supply chain security.
Back to the top
Scanning your dependencies in CI is good practice. I still believe that.
But "we scan" and "we have a control" are different claims, and only one of them survives the question I now ask first when I look at a pipeline: when did this last stop a release?
If the answer is "never", you do not have a gate. You have a report. Reports are fine. Just do not put one on the architecture diagram with a lock icon next to it.