Introduction
We will cover How to Include Files Outside of Docker’s Build Context.
Did you know you can still do this with one simple work-around?
I will break this review in the following sections:
- The problem and limitations of docker
- Show an example use-case of the problem
- Demonstrate a solution with a bonus tip to keep your code intact and still copy outside the directory
I have been using this setup successfully in many of my docker deployments and works like a charm letting me deploy wherever I want without having to maintain multiple files or disorganize my source libraries.
I’m going to keep all the steps simple and straight to the point to avoid you have to spend endless hours debugging or trying to figure out why things aren’t working.
The Problem
The main problem with Docker is that it doesn’t like to honor the COPYing files outside the running context of your application.
Example Of Problem
This off-course results into an issue of not honoring your file structure directories which may be very well organized and you will want to preserve them. Lets look at the following example of my usage I track a repository with Dockerfile’s which is basically a collection of things I have built over the years and want to use them as reference.
~/code/temp/example-project $ ls -al total 8 drwxr-xr-x 3 user staff 96 Feb 6 11:55 . drwxr-xr-x 9 user staff 288 Feb 6 11:51 .. -rw-r--r-- 1 user staff 74 Feb 6 11:55 Dockerfile
While the Dockerfile exists there our source code may live in a separate directory.
If I try to build lets say from an example project and wanting to use those files it would not work. Let’s try to run and this and see what happens:
example-project $ docker build -t example-project . [+] Building 1.1s (7/7) FINISHED => [internal] load build definition from Dockerfile 0.4s => => transferring dockerfile: 116B 0.0s => [internal] load .dockerignore 0.5s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu-dev:latest 0.0s => [internal] load build context 0.3s => => transferring context: 2B 0.0s => [1/3] FROM docker.io/library/ubuntu-dev:latest 0.0s => CACHED [2/3] RUN echo "Hello world" 0.0s => ERROR [3/3] COPY ../permutations.py . 0.0s ------ > [3/3] COPY ../permutations.py .: ------ failed to compute cache key: "/permutations.py" not found: not found
Clearly docker does not like the pathing as it cannot find the proper permutations.py file in the descended path. Lets take a look at how our Dockerfile is written to understand which line failed:
FROM ubuntu-dev:latest RUN echo "Hello world" COPY ../permutations.py .
As you can see the offender is on line 3 where it tries to do the copy.
But lets see if this file exists:
~/code/temp/example-project $ ls -al ../permutations.py -rw-r--r-- 1 user staff 148 Jan 31 10:35 ../permutations.py
The file is clearly there but why is it not working?
Root Cause Of Problem
The root cause of the problem is that docker simply does not like to reference files outside the directory context it’s in. This does not allow you to simply copy over paths from other directories either you reference them relatively or if you reference them explicitly with absolute paths.
Don’t worry though there’s things we can do to fix it. I will break down this below.
How to Include Files Outside of Docker’s Build Context
Now that we understand what the issue is lets go over some solutions on how to fix it and work-around this limitation that Docker has.
The most important part to understand here is that docker likes to reference it’s Dockerfile relatively but keep all your work inside your project organized. Once this is done everything works smoothly with the method we will talk about now.
Solution
The first thing we need to do is bring all of our source code inside the project and then you can keep the Dockerfile wherever you want. In order to do this let’s say we have the source file permutations.py which we tried to copy earlier all we need to do is put it in our root path of project lets say this is a library file so the file structure will now look like this:
~/code/temp/example-project $ mkdir libs ~/code/temp/example-project $ cp ../permutations.py libs ~/code/temp/example-project $ ls -alR -rw-r--r-- 1 user staff 74 Feb 6 11:55 Dockerfile drwxr-xr-x 3 user staff 96 Feb 6 12:06 libs ./libs: -rw-r--r-- 1 user staff 148 Feb 6 12:06 permutations.py
Now that we organized properly our folder structure we can go ahead and do a small adjustment to our Dockerfile in the COPY command.
The file with the changed path will now look like this:
FROM ubuntu-dev:latest RUN echo "Hello world" COPY libs/permutations.py .
The third line is now adjusted to do the copy from the current path.
Let’s re-issue our build command and see if it succeeds this time:
~/code/temp/example-project $ docker build -t example-project . [+] Building 2.8s (8/8) FINISHED => [internal] load build definition from Dockerfile 0.4s => => transferring dockerfile: 118B 0.0s => [internal] load .dockerignore 0.6s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu-dev:latest 0.0s => [internal] load build context 0.2s => => transferring context: 225B 0.0s => [1/3] FROM docker.io/library/ubuntu-dev:latest 0.0s => CACHED [2/3] RUN echo "Hello world" 0.0s => [3/3] COPY libs/permutations.py . 0.5s => exporting to image 1.0s => => exporting layers 0.5s => => writing image sha256:088318504b1e34153f2e2edbf185064e3f8e91feeadd3544516747ae01d031b6 0.0s => => naming to docker.io/library/example-project
Clearly the command now is successful and we have proper docker image.
Bonus – Keep Your Code Structure Intact
One thing you may ask though is what if I have my Dockerfile defined in another project that I just want to inherit and not implement it again inside my code project.
This is a very valid case where you use a reference Dockerfile lets say that can be re-used across different projects and not having to maintain it everywhere separately.
The trick to do this is to leverage the -f flag from the docker command line to accomplish this. To do this we will temporarily move our Dockerfile outside the project directory and try to see if it works by referencing it using -f.
~/code/temp/example-project $ mv Dockerfile ../ ~/code/temp/example-project $ ls -al ../ -rw-r--r-- 1 user staff 76 Feb 6 12:08 Dockerfile ~/code/temp/example-project $ ls -al drwxr-xr-x 3 user staff 96 Feb 6 12:06 libs
As you can see now the example-project directory only contains our libs directory with the source file and the Dockerfile lives outside that context.
If you were to run the docker build command without the -f flag it would fail saying it cannot find the Dockerfile so let’s fix this part too by using the relative specification of the Dockerfile.
~/code/temp/example-project $ docker build -f ../Dockerfile -t example-project . [+] Building 1.6s (8/8) FINISHED => [internal] load build definition from Dockerfile 0.4s => => transferring dockerfile: 36B 0.0s => [internal] load .dockerignore 0.6s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu-dev:latest 0.0s => [1/3] FROM docker.io/library/ubuntu-dev:latest 0.0s => [internal] load build context 0.2s => => transferring context: 66B 0.0s => CACHED [2/3] RUN echo "Hello world" 0.0s => CACHED [3/3] COPY libs/permutations.py . 0.0s => exporting to image 0.4s => => exporting layers 0.0s => => writing image sha256:088318504b1e34153f2e2edbf185064e3f8e91feeadd3544516747ae01d031b6 0.1s => => naming to docker.io/library/example-project
The build succeeded which means we are now able to break out our project as needed into separate directories and still honor the COPY command and it’s functionality.
Conclusion
We were able to successfully demonstrate how to work around the problem of having the COPY command reference files outside the context of our docker build command.
If you found the How to Include Files Outside of Docker’s Build Context useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
What is your favorite docker feature?
My personal favorite is that docker gives me the flexibility to deploy highly customized images and re-use them in my projects.
If you would like to learn more about other Linux and system setup stuff you can check the articles below: