Summary
TL;DR: Solving Jenkins Connectivity Woes in Docker and Kubernetes
-
Issue: Custom configurations in Kubernetes clusters can lead to challenges with Jenkins and Docker interactions.
-
Configuration Snapshot:
- Running Docker natively on a Linux machine with Jenkins installed.
- CI/CD pipeline triggered on code repository updates.
- Jenkins unable to connect to the web frontend, likely due to Docker configurations.
-
Solution:
-
Identify Host IP:
- Run a command to retrieve the host’s IP address.
-
Find Container IP:
- Connect to the Docker container and run a command to obtain its IP address.
-
Firewall Configuration (Optional):
- Optionally, set up firewall rules to allow connections from other Docker containers.
- Note: Adapt the command based on existing firewall rules.
-
Connect Jenkins:
- With the IPs obtained, configure Jenkins to connect to the web frontend successfully.
-
Identify Host IP:
-
Alternate Solution (Not Recommended):
- Use the
--net=host
option when running the web frontend container. - This sets the container’s
localhost
to match the server’s, enabling Jenkins to connect.
- Use the
-
Things Tried That Didn’t Work:
- Attempting to connect to the web application using the host’s IP address.
- Using the DNS record
host.docker.internal
(applicable from Docker 18.03) to connect to containerized applications.
-
Tips:
- Obtain the host’s IP address using a specific command.
- Utilize a bash script to quickly fetch the IP address from a Linux container.
-
Conclusion:
- Follow the provided steps to resolve Jenkins connectivity issues in Docker and Kubernetes setups effectively.
A short form of the solution
Most of the time when you configure your kubernetes cluster, using the default ingress controller settings works. But when you need to do something custom, you may run into problems.
Your underlying docker and kubernetes engine may give you head eaches. We will see in this walkthrough, how to fix on of these and put a smile back on your face.
You need to use the docker0
interface IP of your server. I am running Linux on my server.
ip addr show docker0
You get this answer:
docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::f4d2:49ff:fedd:28a0/64 scope link
valid_lft forever preferred_lft forever
My configuration
I am running docker natively from a Linux machine. I have jenkins installed on it. When a new feature is ready, I push the code to the repository and my CI/CD process get triggered.
It runs jenkins and is suppose to connect to the web frontend to perform some actions.
My problem
Given the configuration above, I need jenkins to be able to connect to the web frontend. But in the this case, jenkins cannot connect.
I believe this is because the configuration inside docker does not allow it.
The solution
Here are the steps that I have used to solve my issue.
Step 1: Get the host’s IP address
Running this command will allow you to grab your host IP address.
ip addr show docker0
You will get this answer:
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::f4d2:49ff:fedd:28a0/64 scope link
valid_lft forever preferred_lft forever
Step 2: Get the container’s IP
Connect to the container and run this command:
ip route show
You get this answer:
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 src 172.17.0.4
Step 3: Setup your firewall to accept the connection from other docker containers (optional)
Running this command does allow connections for my container. Your situation might be different. There might be other rules setup in the firewall. Check the firewall and adapt the command.
iptables -A INPUT -i docker0 -j ACCEPT
Note
Iptables rules are ordered, and this rule may or may not do the right thing depending on what other rules come before it.
Step 4: Connect jenkins and enjoy
Now that everything is setup, I can setup my jenkins and make it connect to the web frontend. Enjoy.
An alternate solution (not recommended)
The idea here is to use the brutal? option --net=host
.
You run the web frontend container with the option --net=host
. This will make the localhost
of the container the same as the localhost of the server.
Jenkins will then be able to connect to the web frontend as if it was connecting to the localhost of the server.
To know more about how to connect jenkins to localhost, check this tutorial.
Few things I have tried and didn’t work
Connect to the host IP directly
I tried to connect to the web application using the host IP address but it failed.
curl http://172.17.1.78:7000/
The answer I got was
curl: (7) Failed to connect to 172.17.1.78 port 7000: No route to host
Use the magic internal address
I have tried it, my apologies. But I encourage you to do so.
From docker 18.03 and above, there is a DNS record that point to your internal IP address host.<a class="wiki-link" href="/blog/en/docker/docker-tip-inspect-and-less">docker</a>.internal
. Use it to connect to your containerized applications.
More information here.
Tips
How to get the IP address of the host
If you need to get the IP address of the host running your docker container, run this command:
ip addr show docker0 | grep -Po 'inet \K[\d.]+'
How to get the container IP address
This quick bash script grabs the IP address from your Linux container.
#!/bin/sh
hostip=$(ip route show | awk '/default/ {print $3}')
echo $hostip