Integrating Security in CI/CD Pipelines: Key Practices

Right now our CI just runs unit tests. We keep saying we’ll add coverage and complexity gates, but every time someone tries, the pipeline slows to a crawl or throws false positives. I’d love a way to enforce basic standards - test coverage > 80%, no new critical issues - without babysitting every PR.

coverage thresholds are overrated if you don’t pair them with meaningful metrics. 80% covered doesn’t mean 80% safe. Half the time, people just write nonsense tests to bump the number. Better to track mutation testing or enforce linting + code smells through SonarQube with fail conditions tuned to your repo’s noise level.

Hi gus, can you share what tools are you using for your CI/CD pipeline? What are the modern best practises you guys follow.

I have been working in Product based company, our tools are nowhere else used except in our org.

Any of you are using Jenkins + Argo + K8S?

Our pipelines take roughly 2 hours to run. I'm migrating our tests, so sometimes, out of the 100k tests, there is 1 or 2 erros, which I fix and push, BUT THEN ANOTHER UNRELATED ERROR SHOWS UP, so I fix that too and push and wait another 2 hours. It's been going like this for 2 days and I'm getting extremely extremely frustrated. just wanted to know if this was common

I work on Plumber, the open-source CI/CD security scanner behind these numbers.

We've been continuously scanning public GitLab and GitHub repos and grading their pipeline configs A to E, Nutri-score style. A single Critical finding (privileged docker-in-docker, curl piped to bash from an untrusted source, that kind of thing) forces an E no matter what else passes.

Across 8,729 public pipelines so far: 63% sit at E. And the findings are not exotic. Unpinned third-party actions, workflows without permission scopes, unprotected default branches, security jobs quietly weakened with allow_failure. All documented for years, all visible in the YAML, none of it caught by a normal code review.

What bugs me is the asymmetry: code gets SAST and SCA, runtime gets DAST and monitoring, but the stage that holds the deploy keys and secrets gets a human skimming a diff, if that. tj-actions and Shai-Hulud both went through exactly that stage. And now AI is writing more of these workflow files than humans review.

• Live data if you want to dig:
• The reasoning behind the grading:
• Grade your own repo with one command (plumber analyze):

Curious how it works elsewhere: who actually owns pipeline config security in your org, and do workflow changes get real review or a rubber stamp?

I'd recommend against hard enforcing coverage - there's too many edge cases that will be annoying to deal with / counterproductive; coverage is a symptom of a cultural problem and needs a cultural solution

as for other issues, run a linter, most linters allow you to configure which kinds of issues should make the linter error (return nonzero exit code) and you can just break the build when that happens; same goes for compiler warnings

This is a follow up of my post on CI/CD best practices from 6 years ago, this time with security angle. Here are the principles:

1. Redundancy: At Least 2 Independent Systems Need to Fail for a Successful Compromise

2. Different Pipelines Must Not Share Credentials

3. Staging Area is a Must

4. Assume Unsafe or Malicious Inputs

5. Pin All Dependencies Consumed by CI

6. Attest, Sign, Verify

Full blog post:

I use Buildkite for most of my stuff. It handles all of my CI needs and I have deployed to Kubernetes, bare metal, ECS, whatever I want it to. Picked it up at a previous job and have been loving it ever since.

Hey Basic_Let7303, yeah sure!

We are a startup, producer repo -> github reusable workflows consumer repos -> app,infra repos

we are basing our cicd in pr events and push to main

1 main orchatrator consuming from reusable workflows in every repo

-pr events are all about quality,security,commit message hygiene, stale branch hygiene

-push to main are all about versioning,building,uploading and updating gitops/running integration/unit/e2e tests

The stack is github for cicd, k8s, argoCD. for security we do sast,sca and image scanning

Picking the tool for the job.

We have Jenkins, GHA, Azure, K8s and VMs

Have to look at each workflow and see what fits best

We are trying to think GHA over Jenkins first, and will start to look at ArgoCD soon, but it a case of the right tool for the job instead of shoe horning it in

Hows everyone doing?

What are some tools you'd recommend that are being widely sought after in production at the moment? I've seen quite the mixed bag of CI/CD tools out there on the hunt for a new role and figured I'd ask here.

I have production experience with Jenkins and Azure DevOps/Pipelines and some personal project experience with GitlabCI (security scanning tools baked into it like Snyk) but I've read that Github Actions and GitlabCI both have some solid left shifted security tools.

Currently, I'm working with AWS, Terraform, Github (Repo), and Bash.I'm looking to add Docker, Kubernetes, and Python to this list. With that said, what CI/CD tooling would you recommend for DevSecOps that would fit nicely within this stack? Also, is there anything you would add to this stack that I should learn that could help get me looked at and considered for more job roles? Lastly, Is there any personal DevSecOps projects you would recommend that would increase my visibility and prepare for interview pipelines?

((I've been actively working on a series of articles that compare and contrast some of these tools as well as how I utilized them for my portfolio to help other DevOps/DevSecOps engineers in the future find work!))

Thank you in advance for reading and your advice!

Hmmmm... For secrets, I would recommend pre-hook commits for SCA. We use Echo for clean images so we need only take care of vulnerabilities that come from the app (so in the commit level we should be good). I would say that before we found that solution, scanning CI/CD would result in too many flags going up and the team would ignore them.

Security guy here. One of the struggles we face in security is implementing vulnerability scans. I could go in depth on the strategy if you want, but at a high level:

We try to move vuln scans into the dev workflow prior to CI. Leverage pre-commit hooks to run local scans early so that the CI flow isn’t interrupted as often. You can use targeted queries here so that there isn’t a lot of unnecessary noise, and you can customize to your org’s needs and risk profile. Your strictest gates should be on your prod branch, explicitly blocking severe (or unacceptable in prod) findings, and there could be less strict gates on lower branches to facilitate speed.

Ultimately though, you need to recognize that adding scans and improved security or code quality scans WILL decrease efficiency. But, leveraging phases and optimizing the scans, along with shifting left to enable devs can help.

Final point - using pre-commit hooks will help remove friction with devs. They don’t need to learn the CLI. They simply learn the output and how to process it, fix the issues, and push their code.

I support it pre-commit/locally via IDE plugins and optional hooks. The idea is that I want devs to be aware of an issue as early as possible. However, there is no guarantee they will do that, so the CI system has to do it as an authoritative check. The pipeline also adds an attestation of the fact that it ran along with the results. This is what allows CD to check and ensure artifacts are authorized for the environment being deployed to.

Let’s see how divided opinions can be on where to run security checks in the development workflow.

I’m talking about things like secrets detection in code, SAST, SCA, among others.

Personally, I see a lot of benefits in running them in the commit:

• Prevents credentials or vulnerable dependencies from ever entering the repo.

• Gives developers instant feedback as the commit is declined.

• Catches issues before they spread into shared branches.

• If the checks are lightweight, the impact on speed is minimal and save CI/CD time later.

That said, post-commit or in the CI/CD pipeline also has its fans, what worked best for you? Where do you run the scans?

By the way, we use commit webhooks in DefendStack, our open-source platform for secrets detection, dependency analysis (SCA) and attack surface management.

If you’re curious or want to contribute, our GitHub repo is: and our Discord community:

Combining both approaches is likely the sweet spot. Use pre-commit hooks for lightweight checks, but let your CI/CD pipeline handle more complex scans, giving you a balanced, efficient workflow without sacrificing developer speed or thorough security testing.

This is not an either/or thing. The quick unobtrusive things can be done in a pre-commit hook (and preferably still in the CI pipeline because hooks are trivial to skip), the rest can be done in a pipeline.

You want to catch things as early as possible, but you also want to make committing as quick and painless a process as possible or people will just tack -n onto their git commit and call it a day. So you limit the pre-commit hooks to the quick unobtrusive stuff and let the pipeline check everything.

Hi everyone,

I’m trying to learn more about real-world DevOps and DevSecOps practices. I’m curious about what companies use in practice, such as:

• CI/CD tools and pipelines
• Best practices for DevOps and DevSecOps
• Design patterns applied in these areas

I’d love to hear your experience and recommendations. Any examples, lessons learned, or tips are greatly appreciated!

If anyone is open to it, I’d be happy to connect and arrange a short meeting to discuss this in more detail.

Thanks in advance!

Both. IDE plugin from your SAST provider, pre-commit hooks to run before commit, and then in CI as part of merge.

Both approaches have trade-offs, but I'd recommend using both or at least combining pre-commit and pre-receive hooks.

Pre-commit hooks are already popular with developers for linting and formatting, so adding secrets detection won't feel disruptive. They'll catch things like .env files before they're committed. I've used gitleaks for this, but the downsides are managing secret regex pattern updates across all developer machines and developers can potentially disable them locally.

Post-commit hooks are easier to deploy universally and catch everything regardless of local dev setup. The problem is once something's committed and pushed, a read-only copy stays in the repo history even if you prevent merging. For non-rotatable secrets like files containing prod customer data, you'd need to force purge the history.

Pre-receive hooks are best of both worlds because they trigger after push but before CI runs. You get full visibility on all commits while maintaining centralized control over detection rules. The catch is this only works with self-hosted solutions like GitHub Enterprise Server, and there's a 5-second timeout limit. GitHub's Enterprise SaaS includes this as part of a larger security bundle, so you can't get just this feature alone.

We've become pretty comfortable putting conventional applications through CI:

• dependency scanning
• SAST
• CodeQL
• secret scanning
• container scanning
• IaC checks
• security policies ...

But what happens when the application being deployed is an AI agent? That may not look particularly interesting in a conventional code diff. But from a security perspective, it could be a significant change.

I'm experimenting with a different CI question:

“What capabilities changed in this PR?”

--

We've implemented an early version of this approach in an open-source static analyzer and connected it to GitHub Actions. (ikaruscareer/SafeAI at GitHub)

The scanner runs locally against the repository and doesn't execute the agent or send the source to a remote service.

I'm curious how other teams approach this.

One thing that I think is really important when threat modeling CI-CD systems is separating your continuous integration from your continuous deployment. Keep all your continuous deployment keys away from all your CI dependencies. We split CI and CD into two separate workflows with different environments in GitHub. Also, attest and Verify 100%. We use cilock.dev for this; it works extremely well with agents like Claude to create SLSA L3 attestations and is FOSS.