Warning on Docker 25 + Portainer/Watchtower

For those using docker + portainer, watch out...

The changes in docker 25.0 (released 1/19) container creation validation makes container creation throw errors [edit: in SOME cases] if the container is using the default/system bridge network (which was quite often the norm pre-docker 25).

Since it errors out on the creation of the container, that means if you update/re-create a container in Portainer it effectively deletes the container (well it deletes it, then tries to create the new one, errors out on creation and leaves you with no container - so effectively deleting it).

If you are using watchtower to auto-update, like many do, you could find yourself with a lot of deleted containers... If they aren't in compose or stacks, well, they're gone and have to be re-created by hand.

In my case I just rolled back my docker VM to a time before that update happened (1/20 on my machine), then fixed all the networking (made a new user bridge network, assigned it to all containers that were previously using the default/system bridge network), then updated again to verify all was good...

But if I wouldn't have had backup to roll back to, well, that would have sucked.

Anyway, FYI / warning.

EDIT:
My guess is that it is this change (but I didn't go double check the pulls):

  • Fix NetworkConnect and ContainerCreate with improved data validation, and return all validation errors at once. moby/moby#46183
2 Likes

I only have portainer and watchtower on the default bridge. Watchtower is deplyed with a stack file using network_mode: bridge in the definition. I just re-deployed the stack and it created it just fine. Maybe the issue only happens if you have no network defined?

If portainer gets an update, watchtower will update it for me... hopefully that does not blow up.

Are you on Docker 25? Jut curious as it would only happen on that version.

If so, then you might be right. I didn't dig in any deeper than understanding it happened to most of my containers that had the system bridge as their specified network.

There may be situations where you can have that defined and it still works, not sure.

All of my stack based containers already had custom user networks, so were unaffected. It was just my one-off containers that got hit.

Here is a rando reddit discussion on it (doesn't have much info though):
https://www.reddit.com/r/OpenMediaVault/comments/19bt03p/getting_an_error_when_trying_to_update_docker/

I think so?

docker-ce-cli/bullseye,now 5:25.0.0-1~debian.11~bullseye amd64 [installed,automatic]
docker-ce/bullseye,now 5:25.0.0-1~debian.11~bullseye amd64 [installed]

I also just checked another stack, which has no network definition but a new bridge was created for it automatically. I think I defined "bridge" for watchtower to prevent it from creating a new bridge, since it is just extra clutter.

This stack has nothing defined for network, its not in the stack file at all. It created its own bridge.

Just re-deployed that one also, seems to be working fine.

I pretty much always use stacks just because it makes it easier for me to change things and re-deploy if needed, and most packages have sample compose files to start with.

1 Like

One more reference, from the portainer side. Not 100% sure it is really a 'bug' as listed, but whatever:

It only happens in certain situations, but is new for Docker 25 so just consider this a raising of awareness.

1 Like

Yeah, I should probably go back and make stacks for a few more of my 'important' containers. I've been lazy on this I guess. Got the containers running how I wanted manually and moved on.

From that reddit post, it sounds like the problem would happen if you are using a config like this
image

but instead of a custom created named network you just leave it as bridge (possibly from the sample run statement or compose file)

That also sort of makes sense based on the error people have posted.

I did not really see many intelligent looking replies to the Reddit post or the portainer bug report besides the one on Reddit pointing me to this guess.

But, for the OP's error, I would start by looking in the compose file for a reference to "bridge." I would start by seeing if you have perhaps a "name: bridge" under the network key in your compose file. Or, if you don't have a separate network key, you may have "networks: -bridge" under your container definitions. Basically, it likely is something where the compose file is written to give the name (alias) "bridge" to a network, which can't be done because "bridge" is a default network not a user-defined one.

I do not think I have any impacted by this but thanks for the head up warning.

1 Like

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.