Reducing Container Image Size by Over 90% and Eliminating 85% of Vulnerabilities: Tackling DevSecOps with Distroless and Trivy

Introduction: The Structural Challenge of "Perfunctory" Vulnerability Scanning

In the operation of container infrastructure, it is common to introduce vulnerability scanning to improve security quality. However, as operations continue, many teams face the challenge of "increased operational load due to noise in scan results."

When using standard base images like ubuntu or debian, a large number of vulnerabilities are detected originating from OS packages (such as curl, git, or old system libraries) that are unrelated to the application code. As a result, the following negative cycle often occurs:

• Limits of Triage: The number of detections becomes enormous (sometimes exceeding 100), burying the application-layer vulnerabilities that actually need to be addressed.
• Delayed Judgment: Even for tools unnecessary in the execution environment, as long as they are detected as vulnerabilities, it takes time to decide whether they can be ignored.
• Becoming Perfunctory: Teams become accustomed to a state where a large number of warnings are constantly issued, leading to a decrease in sensitivity toward security alerts.

This article introduces an approach to break away from this "whack-a-mole" operation by "physically minimizing the attack surface." Specifically, we will share practical methods for sustainable container security by combining the purification of the execution environment through the adoption of Distroless images with the construction of guardrails using Trivy.

Container Vulnerability Scanning

The appeal of containers is their portability—"build once, run anywhere"—but this also means that "vulnerabilities present at the moment of building are packaged and fixed along with it."

In traditional server operations, patches can be applied periodically using yum update or similar commands. However, in containers, vulnerabilities remain unless the image is updated and redeployed.

• Preventing Supply Chain Attacks: Measures against the risk of malicious code being mixed into dependent libraries.
• Reducing Runtime Risk: If shells or unnecessary binaries remain, they can serve as a stepping stone for an attacker to perform "arbitrary operations (shell operations)" if the application is compromised.
• Compliance: In modern system development, leaving OS-level vulnerabilities unaddressed is an audit risk.

Timing of Scans

"Where to run the scan" is a tradeoff between operational robustness and development speed. Generally, checks are performed at the following three timings:

This time, we focused particularly on scanning in CI (GitHub Actions, etc.). The reason is that it serves as a point that functions as a "guardrail to prevent vulnerable images from being pushed to the registry (i.e., preventing contamination before it happens)."

Selection of Scanning Tools

Many tools exist in the field of container scanning, but we selected Trivy by prioritizing the balance between "low operational load" and "wide detection range." A comparison with major tools is as follows:

Reasons for selecting Trivy:

• "All-in-one" Comprehensiveness: While many tools are limited to OS packages and some language libraries, Trivy also supports Dockerfile configuration errors (such as running as root) and IaC scanning for Terraform, etc. The ability to minimize the management cost of using different tools was a major attraction.
• Flexibility as a complete OSS: While commercial tools like Snyk are very powerful, rate limits and license costs must be considered when running a large number of scans in CI/CD. Trivy is OSS, yet its database update frequency is very high, allowing it to be integrated into pipelines without worrying about costs.
• Support for SBOM (Software Bill of Materials): It natively supports the output of SBOMs (CycloneDX / SPDX) required by modern security requirements, providing extensibility toward "asset management" beyond mere vulnerability detection.

Trivy functions not just as a vulnerability detection tool but also as an SBOM generator. This allowed us to decide on its introduction with a view toward not just checking "if there are vulnerabilities now," but also establishing a "system to immediately identify the scope of impact when new vulnerabilities appear in the future (asset transparency)."

Operational Challenges

Once we integrated Trivy into the CI/CD pipeline and established automated scanning, a new structural challenge became apparent: the increase in operational load due to the massive amount of "noise" included in the scan results.

Buried Under Vulnerabilities Originating from OS Packages

For example, when scanning a standard golang:1.22-bookworm (Debian-based) image, more than 80% of the detected vulnerabilities originated from OS packages (libsqlite3, curl, perl, etc.) that are not directly involved in the application's execution.

• Total Detections: Over 3000
• Severity: Includes dozens of Critical and High items

These are included in the base image for "general convenience," but from a security perspective, they become noise that makes it difficult to see the "application-layer vulnerabilities (originating from go.mod) that truly need to be fixed."

Example: Part of the results from running Trivy locally.
Scrutinizing the scan results reveals a mix of two types: OS-derived (debian) and language tool-derived (gobinary). Among them, 2,954 items are detected as vulnerabilities from debian alone.

Report Summary

┌──────────────────────────────────────────────────────────────────────────────────┬──────────┬─────────────────┬─────────┐
│                                      Target                                      │   Type   │ Vulnerabilities │ Secrets │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ vuln-demo:standard (debian 12.9)                                                 │  debian  │      2954       │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/src/cmd/vendor/github.com/google/pprof/third_party/d3flamegraph/pa- │ node-pkg │        0        │    -    │
│ ckage.json                                                                       │          │                 │         │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ app/hello                                                                        │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/bin/go                                                              │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/bin/gofmt                                                           │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/addr2line                                      │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/asm                                            │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/buildid                                        │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/cgo                                            │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/compile                                        │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/covdata                                        │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/cover                                          │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/doc                                            │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/fix                                            │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/link                                           │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/nm                                             │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/objdump                                        │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/pack                                           │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/pprof                                          │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/test2json                                      │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/trace                                          │ gobinary │       22        │    -    │
├──────────────────────────────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤
│ usr/local/go/pkg/tool/linux_arm64/vet                                            │ gobinary │       22        │    -    │
└──────────────────────────────────────────────────────────────────────────────────┴──────────┴─────────────────┴─────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

In particular, many OS packages have no fix patches distributed (Fixed Version is empty), making them "unmanageable noise" that cannot be removed through normal updates.

vuln-demo:standard (debian 12.9)
================================
Total: 2954 (UNKNOWN: 5, LOW: 589, MEDIUM: 1820, HIGH: 530, CRITICAL: 10)

┌────────────────────────────┬─────────────────────┬──────────┬──────────────┬───────────────────────┬────────────────────────┬──────────────────────────────────────────────────────────────┐
│          Library           │    Vulnerability    │ Severity │    Status    │   Installed Version   │     Fixed Version      │                            Title                             │
├────────────────────────────┼─────────────────────┼──────────┼──────────────┼───────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ apt                        │ CVE-2011-3374       │ LOW      │ affected     │ 2.6.1                 │                        │ It was found that apt-key in apt, all versions, do not       │
│                            │                     │          │              │                       │                        │ correctly...                                                 │
│                            │                     │          │              │                       │                        │ https://avd.aquasec.com/nvd/cve-2011-3374                    │
├────────────────────────────┼─────────────────────┤          │              ├───────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ bash                       │ TEMP-0841856-B18BAF │          │              │ 5.2.15-2+b7           │                        │ [Privilege escalation possible to other user than root]      │
│                            │                     │          │              │                       │                        │ https://security-tracker.debian.org/tracker/TEMP-0841856-B1- │
│                            │                     │          │              │                       │                        │ 8BAF                                                         │
├────────────────────────────┼─────────────────────┤          │              ├───────────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ binutils                   │ CVE-2017-13716      │          │              │ 2.40-2                │                        │ binutils: Memory leak with the C++ symbol demangler routine  │
│                            │                     │          │              │                       │                        │ in libiberty                                                 │
│                            │                     │          │              │                       │                        │ https://avd.aquasec.com/nvd/cve-2017-13716                   │
│                            ├─────────────────────┤          │              │                       ├────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2018-20673      │          │              │                       │                        │ libiberty: Integer overflow in demangle_template() function  │
│                            │                     │          │              │                       │                        │ https://avd.aquasec.com/nvd/cve-2018-20673                   │
│                            ├─────────────────────┤          │              │                       ├────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2018-20712      │          │              │                       │                        │ libiberty: heap-based buffer over-read in d_expression_1     │
│                            │                     │          │              │                       │                        │ https://avd.aquasec.com/nvd/cve-2018-20712                   │
│                            ├─────────────────────┤          │              │                       ├────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2018-9996       │          │              │                       │                        │ binutils: Stack-overflow in libiberty/cplus-dem.c causes     │
│                            │                     │          │              │                       │                        │ crash                                                        │
│                            │                     │          │              │                       │                        │ https://avd.aquasec.com/nvd/cve-2018-9996                    │
│                            ├─────────────────────┤          │              │                       ├────────────────────────┼──────────────────────────────────────────────────────────────┤

Triage Costs and the Risk of Perfunctory Scanning

It was not realistic to run the following process for every build against a massive number of detection results:

• Scrutinizing Vulnerabilities: Confirming whether the package is actually used in the execution environment.
• Judging Fixability: Formulating a policy for dealing with vulnerabilities for which no fix patch is provided by the base image (Unfixed).
• Managing Ignore Lists: The maintenance load of exclusion settings using .trivyignore or similar.

As a result, the high volume of alerts caused a decrease in developers' "sensitivity to security," leading to what is known as "alert fatigue." Important warnings were buried in the noise, and signs of the process becoming perfunctory began to appear.

Shifting from "Patching" to "Eliminating the Attack Surface"

Initially, we tried to address this by increasing the frequency of base image updates, but as long as we continued to use general-purpose OS images, there was a limit where several to a dozen unpatched vulnerabilities would always remain upstream.

At this point, we concluded that instead of a symptomatic approach of "fixing vulnerabilities one by one," we needed an architectural solution: "physically eliminating the unnecessary packages themselves, which serve as a breeding ground for vulnerabilities, from the execution environment."

Minimizing Image Size

In a Go application, a shell or package manager is inherently unnecessary in the execution environment. Since it only needs the built binary to run, the combination of "Multi-stage build" and "Distroless" to minimize the execution environment becomes a very powerful solution.

Separating Execution Environments with Multi-stage Builds

First, we completely separate the build environment from the execution environment. For the build, we use a golang image that includes a rich set of tools, and for execution, we use a distroless image where only the binary is copied.

# Stage 1: Build
FROM golang:1.22-bookworm AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Force static linking to create the binary
RUN CGO_ENABLED=0 GOOS=linux go build -o main .

# Stage 2: Runtime
FROM gcr.io/distroless/static-debian12:latest
COPY --from=builder /app/main /main
USER nonroot:nonroot
ENTRYPOINT ["/main"]

Why "Distroless" Instead of "Alpine"?

While alpine is often chosen as a lightweight image, there is a clear difference from a security perspective.

Quantitative Changes After Implementation

As a result of migrating to this configuration, the following significant effects were obtained:

• Image Size: Reduced from 800MB (Standard Debian) to approximately 20MB (Distroless).
• Vulnerability Detections: Decreased from over 100 (including OS package-derived) to just a few (only Go library-derived).

Debugging Techniques in Distroless Environments: Modern Approaches to Compensate for Inconvenience

When introducing Distroless, a concern that always arises from the development floor is "how to troubleshoot in an environment without a shell." Traditional methods of entering the container with docker exec or kubectl exec and investigating with ls or cat can no longer be used.

To resolve this inconvenience, we took the following actions:

Utilizing Kubernetes Ephemeral Containers

By using Ephemeral Containers, which became GA (Generally Available) in Kubernetes 1.25, you can temporarily attach a debug container to a running Pod.

kubectl debug -it <pod-name> --image=busybox --target=<container-name>

With this method, it becomes possible to operate without including any unnecessary tools in the production image, bringing in investigation tools from the outside only when needed.

Using Debug Tags in Parallel

For environments that require frequent investigation, such as development or staging environments, a strategy of separately building debug images that include tools is also effective.

• myapp:latest (Distroless-based / for production)
• myapp:debug (Based on Distroless :debug tag; includes BusyBox shell)

As a preparation for environments where modern kubectl debug cannot be used, creating different images using the Distroless :debug tag is also practical. By using ARG (build arguments) to generate an "image with nothing in it" for production and an "image with an investigation shell" for verification, you can balance development agility with production safety.

ARG RUNTIME_TAG=latest
FROM gcr.io/distroless/static-debian12:${RUNTIME_TAG}
...

Investing in Observability: Knowing the "Inside" from the Outside of the Container

The constraint of "not being able to enter the container" shifts troubleshooting methods from "ad-hoc physical confirmation" to "objective analysis based on data." To achieve this, the following initiatives are indispensable.

1. Thorough Structured Logging

In an environment where you cannot enter the shell, you cannot run cat /var/log/app.log. It is a prerequisite that all logs go "outside."

• Unification to JSON Format: Output logs to standard output (stdout) in JSON format rather than as strings. This enables advanced filtering based on conditions like "specific user ID" or "specific processing time (over 500ms)" in log platforms such as CloudWatch Logs, Datadog, or ELK.
• Adding Context: By including Request IDs or Trace IDs in each log, you can track how a single request was processed in which container without ever entering a shell.

2. Implementing Distributed Tracing

In a microservices environment, it is difficult to identify which service an error occurred in. Introduce OpenTelemetry or similar to visualize the "entire journey" of a request.

• Visualizing Bottlenecks: Even without entering a container to run top, it becomes clear at a glance on a graph which function's processing is taking time or which DB query is lagging.
• Correlation Analysis of Errors: When an error occurs, you can grasp the preceding processing and the response status of dependent external services in chronological order.

3. Resource Monitoring and Profiling

Make the internal state of the binary observable from the outside, not just process liveness monitoring.

• Metrics: For Go, utilize net/http/pprof or Prometheus exporters to monitor memory usage, Goroutine count, GC (Garbage Collection) frequency, etc., in real-time.
• Continuous Profiling: Instead of "entering the container and running pprof," use tools like Datadog Continuous Profiler to constantly collect CPU and memory profiles, allowing for later analysis of low-reproducibility spike phenomena.

"Healthy Operations" Born from Inconvenience

The introduction of Distroless seemed to increase the difficulty of debugging temporarily. However, as a result, the three pillars of "logs, metrics, and traces" were established, and operations evolved from "someone diving into a container to solve it with individual skill" to "the whole team gathering around a dashboard to solve it based on data."

Vulnerability Scanning Without Stopping Development: Integrating Trivy and CI

Even if you introduce a scanning tool, it will become perfunctory as long as it is executed manually. We used GitHub Actions to build a guardrail that "physically prevents vulnerable images from being pushed to the registry."

Configuration of Automated Scans in CI

We integrated Trivy at the end of the build pipeline, running it under the following conditions:

• Filtering by Severity: If there is even one HIGH or CRITICAL item, the build fails and the merge is blocked.
• Using Image Cache: Since downloading the database every time slows down the CI, we optimized it to complete the scan in a few seconds using the GitHub Actions cache function.

- name: Cache Trivy DB
  uses: actions/cache@v4
  with:
    path: .cache/trivy
    key: ${{ runner.os }}-trivy-${{ github.run_id }}
    restore-keys: |
      ${{ runner.os }}-trivy-

- name: Run Trivy vulnerability scanner
  uses: aquasec/trivy-action@master
  with:
    image-ref: 'myapp:${{ github.sha }}'
    format: 'table'
    exit-code: '1' 
    severity: 'HIGH,CRITICAL'
    cache-dir: .cache/trivy

Sorting Necessity of Response to "Keep Development Moving": .trivyignore

If the build stops due to "vulnerabilities for which a fix patch does not yet exist" or "vulnerabilities that are impossible to exploit in the execution environment," development speed will drop significantly. To avoid this, we perform explicit management using .trivyignore.

Operational Rules:

• 1. When ignoring, always leave a comment explaining the reason (why it can be said to be safe).
• 2. Remove from the ignore list as soon as a fix patch is released.

Specific File Description Example

# --- OS Package Noise (False Positives) ---
# libsqlite3: No patch available. No impact; Go binary built without CGO.
CVE-2023-3618

# --- Non-exploitable in Runtime ---
# openssl: Risk mitigated; Go's crypto/tls is used for all external 
# traffic instead of the system openssl.
CVE-2024-0727

Since simply ignoring carries the risk of overlooking security holes, we introduced the following rules:

• Always comment "why it can be ignored": Ensure that the basis for judgment ("no patch provided," "no usage location," etc.) is clear even if read by yourself months later or by another person in charge.
• Include expiration dates or ticket numbers: Create a task in GitHub Issues or similar to "respond when a fix patch is provided" and include that link in the comment.
• Synergy with Distroless: In standard images, this .trivyignore tends to bloat to dozens or hundreds of lines, but with Distroless, since there are almost no OS packages to begin with, this list can be kept to a minimum (a few items).

Obstacles Faced After Implementation and Their Workarounds

When adopting Distroless, especially the minimal static image, you realize that "basic infrastructure components" that were taken for granted in standard images are missing. These can be safely brought in from the build stage using the COPY command in a Multi-stage build.

Time Zone Issues (JST Offset)

Since /usr/share/zoneinfo does not exist in Distroless, running time.LoadLocation("Asia/Tokyo") in a Go program will result in an error. Therefore, copy the time zone data from the build stage (Debian, etc.).

FROM golang:1.22-bookworm AS builder
# ... (Build process) ...

FROM gcr.io/distroless/static-debian12
# Copy timezone data for localized time handling
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
# Set default timezone to JST
ENV TZ=Asia/Tokyo

CA Certificate Issues (External API Communication Failures)

When sending requests to external HTTPS APIs, communication will fail with an SSL/TLS error if there are no root certificates (CA certificates).

Solution: Similarly, copy the latest certificates from the build stage.

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Conclusion

The Limits of "Trying Harder" Operations

The traditional approach of "applying patches one by one" or "adding to the ignore list one by one" for a large number of detected OS vulnerabilities was, so to speak, an effort like "carrying water in a leaky bucket." This "trying harder" leads to the exhaustion of the development team and the overlooking of essential risks (application-layer vulnerabilities).

The Strategic Choice of "Trimming Down"

By adopting Distroless and making the proactive decision to "discard" OS-level management, our focus has shifted as follows:

• Before: Spending our days overwhelmed by triage, living in fear of vulnerabilities found in unused OS libraries.
• After: An environment where we can focus 100% on the areas we are truly responsible for—addressing vulnerabilities in the application layer (e.g., go.mod).

The Role of Infrastructure Engineers: Invisible Guardrails

The "Trivy × CI Integration × Distroless" framework we built isn't about forcing developers to be "constantly security-conscious."

Instead, by laying down invisible guardrails—where vulnerabilities simply cannot enter the pipeline and noise is filtered out—infrastructure engineers allow developers to walk a secure path without even realizing it. This approach improves the security of the entire organization from the bottom up while simultaneously reducing the burden on developers.

If you are exhausted by the daily grind of vulnerability alerts, why not start by "trimming down" your images?