Local and Registry cache not used or *invalidating* cache unnecessarily about buildkit

[tonistiigi]

we noticed that if the source code were to change (like a simple test.txt file being added, or an index.js file being manipulated), our RUN yarn install would no longer be cached.

This is expected with your Dockerfile above that uses COPY .. This will copy different files, resulting in different end build result. Different files would also be visible to the yarn install command so it would need to run again.

Cache mismatches are common if you use COPY . , for example git clone does not create a .git directory in a deterministic way. If you reduce the files used by your build to only files that you actually need, you reduce the chance of cache invalidation.

If you have a reproducible case/steps where you think cache is not matched properly you can post that and we can look at the specifics.

[LucasLundJensen]

This is expected with your Dockerfile above that uses COPY .. This will copy different files, resulting in different end build result. Different files would also be visible to the yarn install command so it would need to run again.

I don't see how the COPY . that is happening after the RUN yarn install step invalidates the previous step?
yarn install is only interacting with with the package.json and yarn.lock file, and running this Dockerfile locally with Docker does not produce the same cache invalidation issues as it does with buildkit.

I tested with pruning all my Docker cache, running it once, adding a test.txt file and running build again, and as shown here, the yarn install is still cached.

=> [internal] load build definition from Dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 261B                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/node:18-alpine                                                                                       0.5s
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                         0.0s
 => [build 1/6] FROM docker.io/library/node:18-alpine@sha256:4837c2ac8998cf172f5892fb45f229c328e4824c43c8506f8ba9c7996d702430                           0.0s
 => [stage-1 1/2] WORKDIR /app                                                                                                                          0.0s
 => [internal] load build context                                                                                                                       0.0s
 => => transferring context: 446B                                                                                                                       0.0s
 => CACHED [build 2/6] WORKDIR /app                                                                                                                     0.0s
 => CACHED [build 3/6] COPY package.json yarn.lock .npmrc /app                                                                                          0.0s
 => CACHED [build 4/6] RUN yarn install --network-timeout 100000                                                                                        0.0s
 => [build 5/6] COPY . /app                                                                                                                             0.0s
 => [build 6/6] RUN yarn build                                                                                                                          1.2s
 => CACHED [stage-1 2/2] COPY --from=build /app/dist /app/dist                                                                                          0.0s
 => exporting to image                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                 0.0s
 => => writing image sha256:4b77c51068465e05ea438134e137de21bbb3c69f287d78e9ffdd81c767df62cf

I'll create a reproducible case early week, just wanted to get initial thoughts out quickly 😊