For I while I’ve had two Docker hosts on my home network, one with Portainer installed and the other one with portainer-agent, to be able to have a single webinterface to manage both.
The one hosting Portainer has always had watchtower installed, in order to automate the upgrade of all containers, the second one did not. Once I realized it, I setup watchtower on it as well, and that’s when issues begun.
The issue
portainer-agent would enter a restart-loop, filling the logs with entries such as:
2021/10/31 07:53:21 [INFO] [main] [message: Agent running on Docker platform] 2021/10/31 07:53:21 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: Error: No such container: 6d3437033dce]
The weird thing is that it was mentioning container 6d3437033dce, while portainer-agent was running as 9bf8b8a94d03.
I suspected it was due to something watchtower was doing when recreating containers after pulling the latest version.
A few searches later, I ended up finding this GitHub issue for portainer-agent, where jackman815 found that the issue was related to the way watchtower assigns the same hostname to the new containers.
I found the problem is about the watchtower, it clones the container configs included hostname when upgrading the container.
When upgrade/re-create the container, by default the docker daemon will assign a new hostname to the container and update internal dns, but the watchtower won’t update the container hostname after the container upgrade so the agent will try to look up the old hostname and it would be getting nxdomain result by docker internal dns.
The solution
As suggested in that issue, the solution is assigning a static hostname to the portainer-agent container.
To do so, stop and delete the old container, then re-create it with the --hostname option.
docker stop portainer_agent docker rm portainer_agent docker run -d -p 9001:9001 --name portainer_agent --hostname portainer_agent --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/volumes:/var/lib/docker/volumes portainer/agent:latest
Of course you’ll have to adjust the container name/hostname to match your setup.