Whatever you gain by running FreeBSD comes at a high cost. And that high cost is keeping FreeBSD jails from taking over.

(I ended up taking another offer but I still think they're onto something.)

We set up a git post receive hook which built static files and restarted httpd on a git receive. Deployment was just 'git push live master'.

While I've used Docker a lot since then, that remains the single easiest deployment I've ever had.

I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.

Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?

Yes. Everything on my box is ephemeral and can be deleted and recreated or put on another box with little-to-no thought. Infrastructure-as-code means my setup is immutable and self-documented.

It's a little more time to set up initially, but now I know exactly what is running.

I don't really understand the 24/7 comment, now that it is set up there's very very little maintenance. Sometimes an upgrade might go askew but that is rare.

Any change to it is recorded as a git commit, I don't have to worry about logging what I've done ever because it's done for me.

Changes are handled by a GitHub action, all I have to do to change what is running is commit a file, and the infra will update itself.

I don't use docker-compose, I use a low-overhead microk8s single-node cluster that I don't think about at all really, I just have changes pushed to it directly with Pulumi (in a real environment I'd use something like ArgoCD) and everything just works nicely. Ingress to services is done through Cloudflare tunnels so I don't even have to port-forward or think about NAT or anything like this.

To update my personal site, I just do a git commit/push, the it's CI/CD builds builds a container and then updates the Pulumi config in the other repo to point to the latest hash, which then kicks off an action in my infra repo to do a Pulumi apply.

Currently it runs on Ubuntu but I'm thinking of using Talos (though it's still nice to be able to just SSH to the box and mess around with files).

I'm not sure why people struggle with this, or the benefits of this approach, so much? It seems like a lot of complexity if you're inexperienced, but if you've been working with computers for a long time, it isn't particularly difficult—there are far more complicated things that computers do.

I could throw the box (old macbook) in a lake and be up and running with every service on a new box in an hour or so. Or I could run it on the cloud. Or a VPS, or metal, or whatever really, it's a completely portable setup.

I still do that for all my personal projects! One of the advantages of docker is that you don't have to rebuild the thing on each deployment target.

https://gist.github.com/bonzini/1abbbdec739e77503945a3605e0e...

There is some dissonance in presenting Podman as a plug-in replacement for Docker, and making it so damn hard to install on (some category's) most popular contemporary LTS Linux distro.

Suddenly you're in a team with 2-3 people and one of them likes to git push broken code and walk-off.

Okay, lets make this less about working with a jack-ass, same setup, but each 5 minutes of downtime cost you millions of dollars. One of your pushes work locally but don't work on the server.

The point of a more structed / complex CI/CD process is to eliminate failures. As the stakes become higher, and the stack becomes more complex, the need for the automation grows.

Docker is just a single part of that automation that makes other things / possible / lowers specific class of failures.

I'm sure what you wrote here is true but i cant fathom how. Maybe its a rh specific issue? (Like how ubuntu breaks rootless bwrap by default)

I'd guess that's because "the spec" is more .jsonschema than a spec about what behaviors any random version should do. And I say "version" because they say "was introduced in version $foo" but they also now go out of their way to say that the file declaring what version it conforms to is a warning

If the software that you're running inside the container supports it, you can use socket activation [0] to get native performance.

[0]: https://github.com/containers/podman/blob/main/docs/tutorial...

Sounds great if you're only running a single web server or whatever. My team builds a fairly complex system that's comprised of ~45 unique services. Those services are managed by different teams with slightly different language/library/etc needs and preferences. Before we containerized everything it was a nightmare keeping everything in sync and making sure different teams didn't step on each others dependencies. Some languages have good tooling to help here (e.g. Python virtual environments) but it's not so great if two services require a different version of Boost.

With Docker, each team is just responsible for making sure their own containers build and run. Use whatever you need to get your job done. Our containers get built in CI, so there is basically a zero percent chance I'll come in in the morning and not be able to run the latest head of develop because someone else's dev machine is slightly different from mine. And if it runs on my machine, I have very good confidence it will run on production.

Not having to regularly rebuild the whole dev environment because I need to work on one particular Python app once a quarter and its build chain reliably breaks other stuff? Priceless.

For me, as an ex-ops, the value proposition is to be able to package a complex stack made of one or more db, several services and tools (ours and external), + describe the interface of these services with the system in a standard way (env vars + mounts points).

It massively simplify the onboarding experience, make updating the stack trivial, and also allow devs, ci and prod to run the same version of all the libraries and services.

If you want to handle all your deployments the same way, you can basically only choose between Nix and containers. Unfortunately, containers are far more popular and have more tooling.

I could get over this. But, IMO, it lends itself to asking the "why" question. Why wouldn't Podman make installing it easier? And the only thing that makes sense to me is that RedHat doesn't want their dev effort supporting their competitor's products.

That's a perfectly reasonable stance, they owe me nothing. But, it does make me feel that anything not in the RH ecosystem is going to be treated as a second-class citizen. That concerns me more than having to build my own debs.

Yes I could use flatpack on ubuntu, however I feel like this is partly something Ubuntu/Debian should provide out-of-the-box

Was this a deal breaker for any company?

I ask because the Docker Desktop paid license requirement is quite reasonable. If you have less than 250 employees and make less than $10 million in annual revenue it's free.

If you have a dev team of 10 people and are extremely profitable to where you need licenses you'd end up paying $9 a year per developer for the license. So $90 / year for everyone, but if you have US developers your all-in payroll is probably going to be over $200,000 per developer or roughly $2 million dollars. In that context $90 is practically nothing. A single lunch for the dev team could cost almost double that.

To me that is a bargain, you're getting an officially supported tool that "just works" on all operating systems.

I have been using it by years. Tested it in Win11 and Linux Mint. I can have even a local kubernetes.

But Docker Engine, the core component which works on Linux, Mac and Windows through WSL2, that is completely and 1000% free to use.

Either the machine is a single security domain, in which case running as root is no issue, or it's not and you need actual isolation in which case run VMs with Firecracker/Kata containers/etc.

Rootless is indeed a world of pain for dubious security promises.

On another note, podman can generate k8s yaml for you, which is a nice touch and easy way to transition.

But verbosity - yeah, kubernetes is absolutely super-verbose. Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each. kubectl really needs some sugar to convert yamls from simple form to verbose and back.

I came across just how slow recently:

- Container -> host: 0.398 Gbps vs. 42.2 Gbps

- host -> container: 20.6 Gbps vs 47.4 Gbps

Source: https://github.com/containerd/nerdctl/blob/main/docs/rootles...

But if you actually add up the time we spend using docker, I'm really not sure it saves that many cycles

https://formulae.brew.sh/formula/podman

With Ubuntu at least, some upstreams publish official PPAs so that you aren't stuck on the rapidly aging versions that Canonical picks when they cut an LTS release.

Debian I found out recently has something similar now via "extrepo".

It can be quite difficult to get this kind of money for such a nominal tool that has a lot of free competition. Docker was very critical a few years ago, but “why not use podman or containerd or…” makes it harder to stand up for.

Either that or you have a massive process to acquire said licenses with multiple reporting requirements. So, you manager doesn’t need the headache and says just use the free stuff and move on.

I used to use docker. I use podman now. Are there teams in my enterprise who have docker licenses - maybe. But tracking them down and dealing with the process of adding myself to that “list” isn’t worth the trouble.

It is for now, but I can't think of a player as large as Docker that hasn't pulled the rug out from under deals like this. And for good reason, that deal is probably a loss leader and if they want to continue they need to convert those free customers into paying.

And usually the need is coming from someone below C-level. So you have to: convince your manager and his manager convince procurement team it has to be in a budget (and usually it's much easier to convince to pay for the dinner) than you have a procurement team than you need to go through vendor review process (or at least chase execution)

This is reality in all big companies that this rule applies to. It's at least a quarter project.

Once I tried to buy a $5k/yr software license. The Sidekiq founder told me (after two months of back and forth) that he's done and I have to pay by CC (which I didn't had as miserable team lead).

If I just want to run a random Docker container, I'm grateful I can get at least "some security" without paying as much in setup/debugging/performance.

Of course, ideally I wouldn't have to choose and the thing that runs the container would be able to run it perfectly securely without me having to know that. But I appreciate any movement in that direction, even if it's not perfect.

But this puts you in a league with some pretty advanced deployment tools, like high level K8, Ansible, cloud orchestration work, and nobody thinks those tools are really that appropriate for the majority of devteams.

People are out here using docker for like... make install.

It seems you never had to deal with timezone-dependent tests.

That said, I'm not a nix guy, but to me, intuitively NixOS wins for this use case. It seems like you could either

A. Use declarative OS installs across deployments B. Put your app into a container which sometimes deploys it's own kernel and then sometimes doesn't and this gets pushed to a third party cloud registry, or you can set up your own registry, and then this container runs on a random ubuntu container or cloud hosting site where you basically don't administer or do any ops you just kind of use it as an empty vessel which exists to run your Docker container.

I get that in practice, these are basically the same, and I think that's a testament to the massive infrastructure work Docker, Inc has done. But it just doesn't make any sense to me

[0]: https://github.com/containers/podman/releases

What else can they do then having a package for every distro?

https://podman.io/docs/installation#installing-on-linux

Including instructions to build from source (including for Debian and Ubuntu):

https://podman.io/docs/installation#building-from-source

I don't know about this specific case but Debian and or Ubuntu having outdated software is a common Debian/Ubuntu problem which nearly always is cause by Debian/Ubuntu itself (funnily if it's outdated in Ubuntu doesn't mean it's outdated in Debian and the other way around ;=) ).

Big companies are made of teams of teams.

The little teams don't really get to make purchasing decisions.

If there's a free alternative, little teams just have to suck it up and try to make it work.

---

Also consider that many of these expenses are born by the 'cost center' side of the house, that is, the people who don't make money for the company.

If you work in a cost center, the name of the game is saving money by cutting expenses.

If technology goes into the actual product, the cost for that is accounted for differently.

You end up having to track who has it installed. Hired 5 more people this week? How many of them will want docker desktop? Oh, we’ve maxed the licenses we bought? Time to re-open the procurement process and amend the purchase order.

It doesn't quite change your argument, but where have you seen $9/year/dev?

The only way I see a $9 figure is the $9/month for Docker Pro with a yearly sub, so it's 12*$9=$108/year/dev or $1080/year for your 10 devs team.

Also it should be noted that Docker Pro is intended for individual professionals, so you don't have collaboration features on private repos and you have to manage each licence individually, which, even for only 10 licences, implies a big overhead.

If you want to work as a team you need to take the Docker Team licence, at $15/month/dev on a yearly sub, so now you are at $1800/year for your 10 devs team.

Twenty times more than your initial figure of $90/year. Still, $1800 is not that much in the grand scheme of things, but then you still have to add a usual Atlassian sub, an Office365/GWorkspace sub, an AI sub... You can end-up paying +$200/month/dev just in software licences, without counting the overhead of managing them.

It is at the company I currently work for. We moved to Rancher Desktop or Podman (individual choice, both are Apache licensed) and blocked Docker Desktop on IT's device management software. Much easier than going through finance and trying to keep up with licenses.

>This section describes how to install Docker Engine on Linux, also known as Docker CE. Docker Engine is also available for Windows, macOS, and Linux, through Docker Desktop.

https://docs.docker.com/engine/install/

I'm not an expert but everything I read online says that Docker runs on Linux so with Mac you need a virtual environment like Docker Desktop, Colima, or Podman to run it.

There are existing tools that fill this gap (Singularity/Apptainer). But, there is always friction when you have to use a specialized tool versus the default. For me, this is a core usecase for rootless containers.

For the reduced feature set we need from containers in bioinformatics, rootless is pretty straightforward. You could get largely the same benefits from chroots.

Where I think the issues start is when you start to use networking, subuids, or other features that require root-level access. At this level, rootless because a tedious exercise in configuration that probably isn’t worth the effort. The problem is, the features I need will be different from the features you need. Satisfying all users in a secure was may not be worth it.

It's only 9 bucks a year, its only 5 bucks a month, its less than a dollar a day.

Docker, ide, ticking system, GitHub, jira, sales force, email, office suit, Figma.... all of a sudden your spending 1000 bucks a month per staff member for a small 10 person office.

Meanwhile AWS is charging you .01xxxx for bandwidth, disk space, cpu time, s3 buckets, databases. All so tiencent based AI clients from China hammer your hardware and run up your bill....

The rent seeking has gotten out of hand.

Which is probably one of the motivations for the blog post. Compatibility will only be there once a large enough share of users use podman that it becomes something that is checked before publish.

1. ssh orb (or machine name if you have multiple) 2. sudo apk add docker docker-cli-compose (install docker) 3. sudo addgroup <username> docker (add user to docker group) 4. sudo rc-update add docker default (set docker to start on startup)

Bonus, add lazydocker to manage your docker containers in a console

1. sudo apk add lazydocker

There's just not much money to be made there, especially considering that docker is a pretty thin wrapper built on top of a bunch of other free technology.

When somebody can make a meaningful (though limited) clone of your core product in 100 lines of bash, you can't make much of a business on top of it [1]

Docker suffers from being useful in the open source space but having no reasonable method to make revenue.

https://github.com/p8952/bocker

In general we do actually try to provide full context for errors from dockerd. Some things can be cryptic because, frankly, they are cryptic and require digging into what really happened (typical of errors from runc), but we do tend to wrap things so at least you know where the call site was.

There's also tracing data you can hook into, which could definitely be improved (some legacy issues around context propagation that need to be solved).

I've definitely seen, in the past, my fair share of errors that simply say "invalid argument" (typically this is a kernel message) without any context but have worked to inject context everywhere or do better at handling errors that we can.

So definitely interested in anything you've seen that could be improved because no one likes to get an error message that you can't understand.

Setting this environment variable helped a lot: KUBECTL_PORT_FORWARD_WEBSOCKETS=true

Note: because Google's quality is falling you won't be able to find this variable using their search, but you can read about it by searching Bing or asking an LLM.

    User=per-service-user
    ExecStart=!podman-wrapper ...

It's really not that hard, folks are just trading Linux knowledge for CI/CD knowledge.

Its React but for DevOps

The podman CLI is nearly a drop-in replacement for docker such that `alias docker=podman` works for many of the most common use cases.

If you don't care about the security implications of running containers as root via a client/server protocol, then by all means keep using Docker. I've switched to podman and I'm happy with my decision, but to each their own.

The container split is often introduced because you have product-a, product-b and infrastructure operations teams/individuals that all share responsibility for an OS user space (and therefore none are accountable for it). You instead structure things as: a host OS and container platform for which infra is responsible, and then product-a container(s) and product-b container(s) for which those teams are responsible.

These boundaries are placed (between networks, machines, hosts and guests, namespaces, users, processes, modules, etc. when needed due to trust or useful due to knowledge sharing and goal alignment.

When they are present in single-user or small highly-integrated team environments, it's because they've been cargo-culted there, yes, but I've seen an equal number of environments where effective and correct boundaries were missing as I've seen ones where they were superfluous.

Almost because most common commands work, but I have not check all.

And almost, because for some docker-compose.yaml which you downloaded/LLM generated you may need to prepend `docker.io/` to the image name

Of course if you have really large / complex compose files or just don't feel like learning something else / aren't using k8s, stick with docker.

Meanwhile, kompose.io exists, which is exactly what it does (but with Go templates as far as I can tell)

> Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each.

Yeah, nothin to add here.

Taking this further (self-plug), you can automatically map your Compose config into a NixOS config that runs your Compose project on systemd!

https://github.com/aksiksi/compose2nix

And of course, Easy =/= Simple, nor the other way around.

or you can use the `build(Layered)Image` to declaratively build an oci image with whatever inside it. I think you can mix and match the approaches.

but yes I'm personally a big fan of Nix's solution to the "works on my machine" problem. all the reproducibility without the clunkiness of having to shell into a special dev container, particularly great for packaging custom tools or weird compilers or other finnicky things that you want to use, not serve.

Nix is, as far as I know, not there and we would probably need weeks of training to get the same result.

Most of the time the value of a solution is not in its technical perfection but in how many people already know it, documentation, and more important all the dumb tooling that's around it!

Are you using rootless podman? Then network redirection is done using user more networking, which has two modes: slirp4netns is very slow, pasta is the newer and good one.

Docker is always set up from the privileged daemon; if you're running podman from the root user there should be no difference.

[1]: https://github.com/apple/container

What you can do if you don't want to use Docker and don't want to maintain these images yourself is have two Podman machines running: one in rootful mode and another in rootless mode. You can, then, use the `--connection` global flag to specify the machine you want your container to run in. Podman can also create those VMs for you if you want it to (I use lima and spin them myself). I recommend using --capabilities to set limits on these containers namespaces out of caution.

Podman Desktop also installs a Docker compatibility layer to smooth over these incompatibilities.

Oh god. I can’t imagine how I could build reliably software if this is what I was doing. How do you know what “patches” are needed to run your software?

If you use k8s for anything, podman might help you avoid remembering yet another iac format.

> Remove layers, keep things simple.

Due to the first line above, I'm not sure if I'm reading the second line correctly. But I'm going to assume that you're referring to the OCI image layers. I feel your pain. But honestly, I don't think that image layers are such a bad idea. It's just that the best practices for those layers are not well defined and some of the early tooling propagated some sub-optimal uses of those layers.

I'll just start with when you might find layers useful. Flatpak's sandboxing engine is bubblewrap (bwrap). It's also a container runtime that uses namespaces, cgroups and seccomp like OCI runtimes do. The difference is that it has more secure seccomp defaults and it doesn't use layers (though mounts are available). I have a tool that uses bwrap to create isolated build and packaging environments. It has a single root fs image (no layers). There are two annoyances with a single layer like this:

1. If you have separate environments for multiple applications/packages, you may want to share the base OS filesystem. You instead end up replicating the same file system redundantly.

2. If you want to collect the artifacts from each step (like source download, extract and build, 'make install', etc) into a separate directory/archive, you'll find yourself reaching out for layers.

I have implemented this and the solutions look almost identical to what OCI runtimes do with OCI image layers - use either overlayfs or btrfs/zfs subvolume mounts.

So if that's the case, then what's the problem with layers? Here are a few:

1. Some tools like the image builders that use Dockerfile/Containerfile create a separate layer for every operation. Some layers are empty (WORKDIR, CMD, etc). But others may contain the results of a single RUN command. This is very unnecessary and the work-arounds are inelegant. You'll need to use caches to remove temporary artifacts, and chain shell commands into a single RUN command (using semicolons).

2. You can't manage layers like files. The chain of layers are managed by manifests and the entire thing needs a protocol, servers and clients to transfer images around. (There are ways to archive them. But it's so hackish.)

So, here are some solutions/mitigations:

1. There are other build tools like buildah and packer that don't create additional layers unless specified. Buildah, a sister project of Podman, is a very interesting tool. It uses regular (shell) commands to build the image. However, those commands closely resemble the Dockerfile commands, making it easy to learn. Thus you can write a shell script to build an image instead of a Dockerfile. It won't create additional layers unless you specify. It also has some nifty features not found in Dockerfiles.

Newer Dockerfile builders (I think buildkit) have options to avoid creating additional layers. Another option is to use dedicated tools to inspect those layers and split/merge them on demand.

2. While a protocol and client/servers are rather inconvenient for lugging images around, they did make themselves useful in other ways too. Container registries these days don't host just images. They can host any OCI artifact. And you can practically pack any sort of data into such an artifact. They are also used for hosting/transferring a lot of other artifacts like helm charts, OPA policies, kubectl plugins, argo templates, etc.

> So any alternative tooling that forces Docker to get it's act together is welcome

What else do you consider as some bad/sub-optimal design choices of Docker? (including those already solved by podman)

This can be a good or a bad thing—good because it's better, but bad because the popularity of Docker sometimes means things aren't compatible and require some tweaking to get running.

This is only partially true. Google's runtime (gvisor) does not share a kernel with the host machine, but still runs inside of a container.

We ditched it for EC2s which were faster and more reliable while being cheaper, but that's beside the point.

Locally I use OrbStack by the way, much less intrusive than Docker Desktop.

You're running into the `/etc/sub[ug]id` defaults. The old default was to start your normal user at 100000 + 64k additional sub-IDs per user, but that changed recently when people at megacorps with hundreds of thousands of employees defined in LDAP and similar ran into ID conflict. Sub-IDs now start at 2^19 on RHEL10 for this reason.

A good reference answer: https://unix.stackexchange.com/questions/651198/podman-volum...

TL;DR: lowercase if a file from the host is shared with a container or a volume is shared between multiple containers. Uppercase in the same scenario if you want the container to take an exclusive lock on the volumes/files (very unlikely).

As others have said, depending on an LLM for this is a disaster because you don’t engage your brain with the manifest, so you aren’t immediately or at least subconsciously aware of what is in that manifest, for good or for ill. This is how bad manifest configurations can drift into codebases and are persisted with cargo-cult coding.

[edit: edit]

Also docker has the network effect. If there was a good light weight tool that was better enough people would absolutely use it.

But it doesn’t exist.

In an ideal world it wouldn’t exist, but we don’t live there.

In the past I think I wound up using https://github.com/mgoltzsche/podman-static because I could not get those podman static binaries to work

(It's titled "Don't Break Debian" but might also be called "Don't Break Ubuntu" as it applies there just as well.)

[1]: https://github.com/microsoft/winget-pkgs/tree/master/manifes...

The majority of businesses in the world, (and the majority of jobs) are created and delivered by small business, not big.

And then the issues when a service goes down it takes everyone else down with it.

A second challenge with the particular setup I’m trying is peer authentication with Postgres, running bare metal on the host. I mount the Unix socket into the container, and on the host Postgres sees the Podman user and permits access to the corresponding DB.

Works really well but only if the container user is root so maps natively. I ended up patching the container image which was the path of least resistance.

https://www.redhat.com/en/blog/generate-selinux-policies-con...

They can do what Docker and many other software providers do that are committed to cross OS functionality. They could build packages for those OSes. Example:

https://docs.docker.com/engine/install/ubuntu/#install-using...

The install instructions you link to are relying on the OS providers to build/package Podman as part of their OS release process. But that is notoriously out-of-date.

You could argue, "Not Podman's Problem", and, in one sense, you'd be right. But, again, it leads to the question "Why wouldn't they make it their problem like so many other popular projects have?" and I believe I answered that previously.

An IT department for a company of that size should have ironed out workflows and automated ways to keep tabs on who has what and who needs what. They may also be under various compliance requirements that expect due diligence to happen every quarter to make sure everything is legit from a licensing perspective.

Even if it's not automated, it's normal for a team to email IT / HR with new hire requirements. Having a list of tools that need licenses in that email is something I've seen at plenty of places.

I would say there's lots of other tools where onboarding is more complicated from a license perspective because it might depend on if a developer wants to use that tool and then keeping tabs on if they are still using it. At least with Docker Desktop it's safe to say if you're on macOS you're using it.

I guess I'm not on board with this being a major conflict point.

I do understand that this mostly is because management wants staff to be replaceable and disposable having specialty tools suggests that a person can be unique.

I don't quite get this argument. How is that different from any piece of software that an employee will want in any sort of enterprise setting? From an IT operations perspective it is true that Docker Desktop on Windows is a little more annoying than something like an Adobe product, because Docker Desktop users need their local user to be part of their local docker security group on their specific machine. Aside from that I would argue that Docker Desktop is by far one of the easiest developer tools (and do note that I said developer tools) to track licenses for.

In non-enterprise setups I can see why it would be annoying but I suspect that's why it's free for companies with fewer than 250 people and 10 million in revenue.

It's not just that you need a licence now, it's that even if we took it to procurement, until it actually got done we'd be at risk of them turning up with a list of IP addresses and saying "are you going to pay for all of these installs, then?". It's just a stupid position to get into. The Docker of today might not have a record of doing that, but I wouldn't rule out them getting bought by someone like Oracle who absolutely, definitely would.

(base) kord@DESKTOP-QPLEI6S:/mnt/wsl/docker-desktop-bind-mounts/Ubuntu/37c7f28..blah..blah$ podman

Command 'podman' not found, but can be installed with:

sudo apt install podman

I can't see how any kind of sensible security evaluation process would reach that conclusion. If you trust your users you don't need rootless, if you don't trust your users rootless containers aren't good enough. I suspect people do rootless because it seems easy and catches a few accidental mistakes rather than it being a legitimate security measure.

(Should live at https://forgejo.org/docs/v12.0/admin/actions/docker-access/ once it is finished up, if anyone runs into the comment after the draft is gone.)

Most likely gp is having issues with volumes and hasn’t figured out how to mix the :z and :Z attribute to bind mounts. Or the containers are trying to do something that security-wise is a big no-no.

In my experience SELinux defaults have been much wiser than me and every time i had issues i ended up learning a better way to do what i wanted to do.

Other than that… it essentially just works.

The benefit is that, Alpine has access to all your local and network drives so you can use them. You can sandbox them as well. It's not a big learning curve, just a good VM with access to all drives but isolated to local only.

That's good to know it works well for you, because I would prefer not to use docker.

1. People hear about how great rootless is with Podman but then expect to be able to switch directly from rootful Docker to rootless Podman without changing anything. The only way that could work is if there was no difference between rootful and rootless to begin with, but people don't want to hear that. They combine these two selling points in their head and think they can get both a drop-in replacement for Docker and also rootless by default. The proper choice is to either switch from rootful Docker to rootful Podman *or* put in the work to make your container work in rootless, work you would also have had to do with rootless Docker.

2. Docker Compose started out as an external third-party add-on (v1) which was later rewritten as an internal facility (v2) but `podman compose` calls out to either `docker-compose` (i.e. v1) or to its own clone of the same mechanism, `podman-compose`. The upshot is a lot of impedance mismatch. Combine that with the fact that Podman wants you to use Quadlets anyway, resulting in less incentive to work on these corner cases.

3. Docker has always tried to pretend SELinux doesn't exist, either by hosting on Debian and friends or by banging things into place by using their privileged (rootful) position. Podman comes from Red Hat, and until recently they had Mr SELinux on the team. Thus Podman is SELinux-first, all of which combines to confuse transplants who think they can go on ignoring SELinux.

4. On macOS and Windows, both Podman and Docker need a background Linux VM to provide the kernel, without which they cannot do LXC-type things. These VMs are not set up precisely the same way, which produces migration issues when someone is depending on exact details of the underlying VM. One common case is that they differ in how they handle file sharing with the host.

Colima would/should never be used in production for a number of reasons, but yeah it's great for local only development on a laptop.

https://cloud.google.com/blog/products/serverless/cloud-run-...

I’ve dealt with a fair bit of Swarm internals for https://lunni.dev/, and I’m ready to switch to k8s any moment. Don’t wanna lose Compose Spec support though, so I’ve started an IaC thingy that can map it to both k8s and Swarm resources. (Now I need to find some time for it!)

/s

See sibling comment above about them being just remote binaries.

[0]: https://github.com/containers/podman/blob/c8183c50/Makefile#...

[1]: https://github.com/containers/podman/releases

I'm in IT consulting. If most companies could even get the basic best practices of the field implemented, I wouldn't have a job.

Large companies do have ways to deal with this: they negotiate flat rates or true-up cadences with vendors. But now you’ve raised the bar way higher than “just use podman”.

The business world is full of things that "should" be a certain way, but aren't.

For the technology world, double the number.

We'd all like to live in some magical imaginary HN "should" world, but none of us do. We all work in companies that are flawed, and sometimes those flaws get in the way of our work.

If you've never run into this, buy a lottery ticket.

The usual way that procurement is handled, for the sake of everybody's sanity, is to sign a flat-rate / tiered contract, often with some kind of true-up window. That way the team that's trying to buy software licenses doesn't have their invoices swinging up/down every time headcount or usage patterns shifts, and they don't have to go back to the well every time they need more seats.

This is a reasonably well-oiled machine, but it does take fuel: setting up a new enterprise agreement like that takes humans and time, both of which are not free. So companies are incentivized to be selective in when they do it. If there's an option that requires negotiating a license deal, and an option that does not, there's decent inertia towards the latter.

All of which is a long way to say: many large enterprises are "good" at knowing how many of their endpoints are running what software, either by making getting software a paperwork process or by tracking with some kind of endpoint management (though it's noteworthy that there are also large enterprises that suck at endpoint management and have no clue what's running in their fleet). The "hard" part (where "hard" means "requires the business to expend energy they'd rather not) is getting a deal that doesn't involve the license seat counter / invoice details having to flex for each individual.

It feels a little hypocritical for us to feed our families through our tech talent and then complain that someone else is doing the same.

I get the why most people think they need containers, but it really seems only suited for hyper-complex (ironically, Google) deployments with thousands of developers pushing code simultaneously.

https://cloud.google.com/blog/products/serverless/cloud-run-...

Though as someone who's used a lot of Azure infrastructure as code with Bicep and also done the K8s YAML's I'm not sure which is more complicated at this point to be honest. I suspect that depends on your k8s setup of course.

Containers are the packaging format, EC2 is the infrastructure. (docker, crio, podman, kata, etc are the runtime)

When deploying on EC2, you still need to deploy your software, and when using containers you still need somewhere to deploy to.

I guess that depends on how many you need to do

BTW, I'm talking about docker/compose files. kubectl doesn't have a conversion there. When converting from podman, it's super simple.

Docker would be wise to release their own similar tool.

compose syntax isn't that complex, nor would it take advtange of many k8s features out of the box, but it's a good start for a small team looking to start to transition platforms

(Have been running k8s clusters for 5+ years)

Docker and other containerization solved the “it works on my machine” issue

providing duplicate/additional non official builds for other OS is

- undermining the OSes package curation

- confusing for the user

- cost additional developer time, which for most OSS is fairly limited

- for non vendorable system dependencies this additional dev time cost can be way higher in all kinds of surprising ways

- obfuscate if a Linux distro is in-cable of properly maintaining their packages

- lead to a splitting of the target OS specific eco system of software using this as a dependency

etc.

it's a lose lose lose for pretty much everyone involved

so as long as you don't have a have a monetary reason that you must do it (like e.g. docker has) it's in my personal opinion a very dump thing to do

I apologize for being a bit blunt but in the end why not use a Linux distribution which works with modern software development cycles?

Blaming others for problems with the OS you decided to use when there are alternatives seems not very productive.

Correct, but every additional software package and each additional license adds more to track.

Every new software license requires legal to review it.

These centralized departments add up all of the license and SaaS costs and it shows up as one big number, which executives start pushing to decrease. When you let everyone get a license for everything they might need, it gets out of control quickly (many startups relearn this lesson in their growth phase)

Then they start investigating how often people use software packages and realize most people aren't actually using most software they have seats for. This happens because when software feels 'free' people request it for one-time use for a thing or to try it out and then forget about it, so you have low utilization across the board.

So they start making it harder to add new software. They start auditing usage. They may want reports on why software is still needed and who uses it.

It all adds up. I understand you don't think it should be this way, but it is at big companies. You're right that that the $24/user per month isn't much, but it's one of dozens of fees that get added, multiplied by every employee in the company, and now they need someone to maintain licenses, get them reviewed, interact with the rep every year, do the negotiation battles, and so on. It adds up fast.

Also, I don't want to have to troubleshoot why the docker daemon isn't running every time I need it

1. You want to control spend - there are budgets. 2. You want to control accounting - minimize the number of vendors you work with. Each billing needs to come with an invoice, these need to be managed, when a developer leaves you need to cancel their seat etc. It's a pain. 3. You want to control compliance - are these tools safe? Are they accessing sensitive data? Are they audited? 4. You want to control interoperability between teams. Can't have it become a zoo of bring-your-own stuff.

So free tools get around all of these, you can just wing it under the radar and if the tool becomes prominent enough then you go fight the war to have it adopted. Once there's spend, you need to get into line. And that line makes a lot of sense when you're into 30 developers, let alone hundreds.

I don't think there's any stiffing going on, since the open source contributors knowingly contributed with a license that specifically says that payment isn't required. It is not reasonable for them to take the benefits of doing that but then expect payment anyway.

[1]:https://github.com/containers/podman-compose/issues/792

These are almost always multi-tennant with differing levels of trust and experience between users. The data processed here can often have data access agreements or laws that limit who can see what data. You can’t have a poorly configured container exposing data, for example. So, the number of people who have root access is very limited. Normal users running workflows would all be required to run code rootless.

My point is: If figuring things out with podman is similar to my experience, I understand why people don't want to do that. Do they have a definitive page dedicated to setting up Selinux for podman, that is well maintained and guaranteed to solve all Selinux issues, and allows me to use bind mounts with readonly permission?

I'm using dind too, but this requires privileged runners...

we had some similar issues and it was due to containers running out of resources (mainly RAM/memory, by a lot, but only for a small amount of time). And it happens that in rootless this was correctly detected and enforced, but on non rootless docker (in that case on a Mac dev laptop) it didn't detect this resource spikes and hence "happened to work" even through it shouldn't have.

You can also use this to create a VM for Podman that runs on Fedora, rootful by default: https://github.com/carlosonunez/bash-dotfiles/blob/main/lima...

If you go the Lima approach, use `podman system connection add` to add rootful and rootless VMs, then use the `--connection` flag to specify which you want to use. You can alias them to make that easier; for instance, use `alias podman=podman` for rootless stuff (assuming the rootless VM is your default) nad `alias rpodman=podman --connection rootful` for rootful stuff. I'll write a post describing how to set all of that up soon!

Costs and management grow in an O(n*m) manner where n is employees and m is numbers of licenses per employee. It seems like nothing when you're small and people only need a couple licenses, but a few years in the aggregate bills are eye-popping and you realize the majority of people don't use most of the licenses they've requested (it really happens).

Contrast this with what it takes for an engineer to use a common, free tool: They can just use it. No approval process. No extra management steps for anyone. Nothing to argue that you need to use it every year at license audit time. Just run with it.

Open source is different in exactly that, no procurement.

Finance makes procurement annoying so people are not motivated to go through it.

1. As parallel commenters have pointed out, no. Plenty of open source developers exist who aren't interested in getting paid for their open source projects. You can tell this because some open source projects sell support or have donation links or outright sell their open source software and some do not. This line of thinking seems to come out of some utopian theoretical world where open source developers shouldn't sell their software because that makes them sell-outs but users are expected to pay them anyways.

2. I do love the idea of large companies paying for open source software they use because it tends to set up all kinds of good incentives for the long term health of software projects. That said, paying open source projects tends to be comically difficult. Large companies are optimized for negotiating enterprise software agreements with a counterparty that is primed to engage in that process. They often don't have a smooth way to like, just feed money into a Donate form, or make a really big Github or Patreon Sponsorship, etc. So even people in large companies that really want to give money to open source devs struggle to do so.

I'm not sure you realize that "open source" means anyone anywhere is free to use, modify, and redistribute the software in any way they see fit? Maybe you're thinking of freeware or shareware which often _do_ come with exceptions for commercial use?

But anyway, as an open source contributor, I have never felt I was being "stiffed" just because a company uses some software that I helped write or improve. I contribute back to projects because I find them useful and want to fix the problems that I run into so I don't have to maintain my own local patches, help others avoid the same problems, and because making the software better is how I give back to the open source community.

and sharing files from the host, ide integration, etc.

Not that it can't be done. But doing it is not just, 'run it'. Now you manage a vm, change your workflow, etc.

Comparing root docker with rootless podman performance is apples to oranges. However, even for rootless pasta does have good performance.

There are many benefits to be had for individuals and small companies as well. The piece of mind that comes with immutable architecture is incredible.

While it's true that you can often get quite far with the old cowboy ways, particularly for competent solo devs or small teams, there's a point where it starts to unravel, and you don't need to be a hyper-complex mega-corp to see it happen. Once you stray from the happy path or have common business requirements related to operations and security, the old ways become a liability.

There are reasons ops people will accept the extra layers and complexity to enable container-based architecture. They're not thrilled to add more infrastructure, but it's better than the alternative.

Have a look at https://kompose.io/ as well.

This is going to differ company to company but since we're narrowing it to large companies I disagree. Usually there's a TPM that tracks license distribution and usage. Most companies provide that kind of information as part of their licensing program (and Docker certainly does.)

> Every new software license requires legal to review it.

Yes, but this is like 90% of what legal does - contract review. It's also what managers do but more on the negotiation end. Most average software engineers probably don't realize it but a lot of cloud services, even within a managed cloud provider like AWS, require contract and pricing negotiation.

> These centralized departments add up all of the license and SaaS costs and it shows up as one big number, which executives start pushing to decrease. When you let everyone get a license for everything they might need, it gets out of control quickly (many startups relearn this lesson in their growth phase)

As I said earlier, I can't speak for other companies but at large companies I've worked at this just simply isn't true. There's metrics for when the software isn't being used because the corporation is financially incentivized to shrink those numbers or consolidate on software that achieves similar goals. They're certainly individually tracked fairly far up the chain even if they do appear as a big number somewhere.

Also, latest with 20 employees or computers, someone in charge of IT (sysadmin, IT department) would decide to use a software asset management tool (aka software inventory system) to automatically track, roll out, uninstall, monitor vetted software. Anything else is just unprofessional.

Or when your IT department is prohibited from purchasing anything that doesn't come from Microsoft or CDW.

https://docs.orbstack.dev/features/debug

Let alone the local resource monitor, increased, performance, automated local domains (no more complicated docker network settings to get your app working with local host), and more.

kompose: https://kubernetes.io/docs/tasks/configure-pod-container/tra...

Also, technically docker-compose was the first orchestration tool compared to Kubernetes. Expecting former to provide a translation layer for the latter is rather unorthodox. It is usually the latter tool provides certain compatibility features for former tools...

Kustomize eliminates the vast majority of the duplication (i.e. a unique fact about the cluster being expressed in more than one place), it's just the boilerplate that's annoying.

The base image building can be pretty easily automated, then individual projects using those base images can expect new base images on a regular basis, and test updating to the latest at their leisure without getting any surprise changes.

Mostly agree. But something like Podman w/ RedHat behind it is unlikely to be limited in the same way a lot of community OSS projects are.

Unfortunately, I disagree with just about every other point you made but don't think it's worth responding point-by-point. In short, I think a project having dedicated builds for popular OSes is a win-win for just about everyone, excepting that it does take, sometimes a considerable amount of, effort to support those cross OS builds. Additionally, there are now options like Snap/Flatpack/AppImage that can be targets instead of the OS itself, although there is admittedly a tradeoff there as well.

For some projects, say something like ripgrep, just using what is in the OS repo is fine because having the latest and greatest features/bug-fixes is unlikely to matter to most people using the tool.

But, on something like Podman, where there is so many pieces, it's a relatively new technology, and the interaction between Kernel, OS, and user space is so high, being stuck with a non-current OS provided release for a couple years is a non-starter.

> why not use a Linux distribution which works with modern software development cycles?

Because I like my OS to be stable, widely supported, and I also like some of my applications to be evergreen. I find Ubuntu is usually a really good mix that way and I'm going on 15+ years of use. There are other solutions for that that I could use, but I'm mostly happy where I am and don't want to spend the kind of time it would take to adopt a different OS and everything that would follow from that.

That leads _me_ to avoid Podman currently. I can appreciate that you have a different opinion, I just think you are a overplaying your perspective a bit in the comment above.

But I have to feed my family.

I've worked at companies that size and the "war" involved putting time in the calendar of the head of engineering, asking how his son was, demoing the product we wanted for about two minutes and explaining the pain point it solved, then promising to get our legal team and the one security person to review it after he put the credit card in and before we used it in prod. When I worked somewhere larger it was much more difficult.

Where do you work ? Is that even possible in 2025?

Not that this should be an argument for docker. The idea that having someone to call makes a piece of software "safer" is as ridiculous at it sounds. Especially if you've ever tried "calling" a company you buy 20 licenses from, and when I say call what I really mean is talking with a chatbot and then waiting a month for them to get back to you via email. But IT's gonna IT.

As far as IT operations goes, it's usually easier to get approval for paid products since they come with support and are viewed as more "trustworthy". At least in my experience.

I've never worked in a 300+ organisation where you could "just use" things. I have worked in places where they gave some of us local admins (I've been a domainadmin in a few places too), but there is usually a large bureaucracy around software regardless of licenses. Where I work right now, licensing is a minor part of it for companies with good payment systems (like Docker) where it'll automatically go on the books and be EU tax deducted. Compare that to GitKraken where you need to create an IT owner account inside their system, and then distribute the annual licenses manually after you pay for them with a credit card that you will then need to manually submit for tax deduction.

To draw a parallel: imagine a large open source project with a large userbase. The users interact with the project and a bunch of them have ideas for how to make it better! So they each cut feature requests against the project. The maintainers look at them. Some of the feature requests they'll work on, some of them they'll take well-formed pull requests. But some they'll say "look, we get that this is helpful for you, but we don't think this aligns with the direction we want the project to go".

A good procurement team realizes that every time the business inks a purchase agreement with a vendor, the company's portfolio has become incrementally more costly. For massive deals, most of that cost is paid in dollars. For cheaper software, the sticker price is low but there's still the cost of having one more plate to juggle for renewals / negotiations / tracking / etc.

So they're incentivized to be polite but firm and push back on whether there's a way to get the outcome in another way.

(this isn't to suggest that all or even most procurement teams are good, but there is a kernel of sanity in the concept even though it's often painful for the person who wants to buy something)

Having used Docker Desktop on a Mac myself, it seems... fine? It does the job well enough, and it’s part of the development rather than production flow so it doesn’t need to be perfect, just unobtrusive.

https://tangentsoft.com/podman/wiki?name=Not%20a%20Drop-In%2...

Thanks!

https://tangentsoft.com/podman/wiki?name=Not%20a%20Drop-In%2...

[1]. https://docs.podman.io/en/latest/markdown/podman-system-serv...

It's much more than a gui for it supports running k8s locally, managing custom vm instances, resource monitoring of containers, built in local domain name support with ssl mycontainer.orb, a debug shell that gives you ability to install packages that are not available in the image by default, much better and automated volume mounting and view every container in finder, ability to query logs, an amazing ui, plus it is much, much faster and more resource efficient.

I am normally with you that terminal is usually enough, but the above features really do make it worth it especially when using existing services that have complicated failure logs or are resource intensive like redis, postgres, livekit, etc or you have a lot of ports running and want to call your service without having to worry about remembering port numbers or complicated docker network configuration.

I basically use (orbstack) docker containers as light weight VM, easily accessible through multiple shells and they shutdown when nothing is running anymore.

I use them for development isolation? Or when I need to run some tool. It mounts the current directory, so your container is chrooted to that project.

I would use containers too, in such cases.

sure I agree that where it's easily doable (like e.g. ripgrep) having non distro specific builds is a must have

But sadly this doesn't fully work for podman AFIK as it involves a lot of subtle interactions with things which aren't consistently setup across Linux distros with probably the worst offender being the Linux security modules system (e.g. SELinux, AppArmor etc.). But thinking about probably sooner or later you probably could have a mostly OS independent postman setup (limited to newer OS versions). Or to be more specific 3 one with SELinux one with AppArmor and neither with neither, so I guess maybe not :/

exactly. I've built podman for debian. It's not an esoteric target. It gets a little hairy with all of the capabilities stuff and selinux, but it's feasible. Give me, I don't know, $10k a quarter and I'd probably do it.

After many years of "I want stability and evergreen", I finally realized that this is Fedora. Each release is very stable, and they arrive more often than once an eon.

Or so it seems to me whenever I have to deal with them. We ended up with Microsoft defender on our corp Macs even.. :|

There is no bottom to the barrel, and incompetence and insensitivity can rise quite high in some cases.

Idk what the problem is, but it's ugly. I switched to orbstack because there was something like a memory leak happening with docker desktop, just using waaaaay too many resources all the time, sometimes it would just crash. I just started using docker desktop from the get-go because when I came on I had multiple people with more experience say 'oh, you're coming from linux? Don't even try to use the docker daemon, just download docker desktop'.

Because it's not very useful by itself for running production infra, but it's great for helping to develop it.

Otherwise you're going to see more and more move to podman (and podman desktop) / OCI containers over time, as corps won't have to pay the docker tax and will get better integration with their existing k8s platform.

[1] https://hpc-wiki.info/hpc/Modules

What could possibly go wrong?

And then there's the windowing system of macOS that feels like it's straight from the 90s. "System tray" icons that accumulate over time and are distracting, awful window management with clunky animations, the near useless dock (clicking on VS Code shows all my 6 IDEs, why?). Windows and Linux are much modern in that regard.

The Mac hardware is amazing, well worth its price, but the OS feels like it's from a decade ago.

I use WSL for work because we have no linux client options. It's generally fine, but both forced windows update reboots as well as seemingly random wsl reboots (assuming because of some component update?) can really bite you if you're in the middle of something.

But it is not cross-platform, so we settled on Podman instead, which came (distant) second in my tests. The UI is horrible, IMO but hey… compromises.

I use OrbStack for my personal stuff, though.

i've been using an archlinux vm for everything development over the past year and a half and i couldn't be happier.

It costs about $100/year per seat for commercial use, IIRC. But it is significantly faster than Docker Desktop at literally everything, has a way better UI, and a bunch of QoL features that are nice. Plus Linux virtualization that is both better and (repeating on this theme) significantly more performant than Parallels or VMWare Fusion or UTM.

The above features really do make it worth it especially when using existing services that have complicated failure logs or are resource intensive like redis, postgres, livekit, etc or you have a lot of ports running and want to call your service without having to worry about remembering port numbers or complicated docker network configuration.

Check it out https://docs.orbstack.dev/

If you're running rootless Podman containers then the Podman API is only running with user privileges. And, because Podman uses socket activation, it only runs when something is actively talking to it.

On Linux, for development, podman and docker are pretty similar but I prefer the k8s yaml approach vs compose so tend to use podman.

I don't think Apple really cares about dev use cases anymore so I haven't used a Mac for development in a while (I would for iOS development of course if that ever came up).

What you say is absolutely correct. If Docker keeps creating compatibility layers for it's competitors, it makes everyone to switch to a competitor. In this case, the competitor is Kubernetes as it's running in production for much larger scale (enterprise workloads) compared to Podman et. al.

Hence, it's the job of Podman, Kubernetes, et. al. to write their compatibility layer to provide a value-add for their customers.

The new docs split that out into separate podman-container/volume/etc.unit(5) pages, with quadlet.7 being the index page. So they're still linking to the same documentation, just the organization happened to change underneath them.

If you must see what they linked to originally, the versions docs are still the original organization (i.e. all on one page): https://docs.podman.io/en/v5.6.0/markdown/podman-systemd.uni...

Or so I was told when I made the monumental mistake of trying to fight such a policy once.

So now we just have a don't ask don't tell kind of gig going on.

I don't really know what the solution is, but dev laptops are goldmines for haxxors, and locking them down stops them from really being dev machines. shrug

If you're building really arch-specific stuff, then I could see not wanting to go there, but Rosetta support is pretty much seamless. It's just slower.

    zips the local copy of the branch and rsyncs it to the environment, and some other stuff

You're basically splitting the process of building and distributed your application into: write the software, build the image, deploy the image.

Everyone who uses these tools, which is most people by this point, will understand these steps. Additionally, any framework, cloud provider, etc that speaks container images, like ECS, Kubernetes, Docker Desktop, etc can manage your deployments for you, since they speak container images. Also, the API of your container image (e.g. the environment variables, entrypoint flags, and mounted volumes it expects) communicate to those deploying your application what things you expect for them to provide during deployment.

Without all this, whoever or whatever is deploying your application has to know every little detail and you're going to spend a lot of time writing custom workflows to hook into every different kind of infrastructure you want to deploy to.

it's pretty stupid because the same curl | bash that could have done that could have just posted the same contents directly to the internet without the container. The best chance you actually have is to do as much development as possible inside a sealed environment like ... a container where at least you have some way to limit visibility of partially trusted code of your file system.

On the contrary, docker documentation *is* stable, I had bookmarks from 10-years ago on the *latest* editions, that still work today. The final link may have changed, but at least, there is a redirect (or a text showing that has been moved) instead of plain 404/not-found.

This is a crucial part of the quality applications offer. There might've been 100s of podmans probably since Docker was launched more than 10 years ago, but none came close to maintain high-quality of documentation and user-interface (ie. cli commands, switches). Especially in the backward-compatible way.

It's a different style of documentation organization: if you want to link to a specific version you should link to the specific version not latest. I won't argue it's necessarily a better way of doing things than Docker, but knowing it's the same thing as what's with the package is nice.

Just put this thread into a whatever LLM. I overall see 2 major themes here. Compatibility and stability issues, all over the place. Not just documentation, but with other tools. Compose schema v2 does not match the current/latest one, missing functionality (although this one is acceptable at certain level), etc.

Also, as soon as the docs were "posted", it became obsolete/useless/deprecated. I mean, what sort of quality are we talking about here?