PostgreSQL 16 needed for GitLab 18

[mihelm]

The blog post on the release of GitLab 17.11 contains this bit:

In GitLab 18.0, the minimum-supported version of PostgreSQL will be version 16. To prepare for this change, on instances that don’t use PostgreSQL Cluster, upgrades to GitLab 17.11 will attempt to automatically upgrade PostgreSQL to version 16.

What are the prospects for a sameersbn/postgresql image with version 16?

[pmoralesp]

Just updated to 17.11.0 and broke my setup because of this

[th-2021]

The upcoming release page mentions: 18.0 (2025-05-15)

[kkimurak]

I think this should probably be discussed in the sameersbn/docker-postgresql repository, not here (I don't know if there will be a response from the maintainers early on, i.e. before the release of gitlab 18 when postgresql16 becomes mandatory..)

Anyway, as a temporary workaround, I forked sameersbn/postgresql and pushed kkimurak/sameersbn-postgresql I built myself. Currently :15 (:15.12, :15.12-1) and :16 (:16.8, :16.8-1) are available. I have confirmed that the scheduled backup using :16.8-1 works properly in combination with sameersbn/gitlab:17.11.0.
See https://hub.docker.com/r/kkimurak/sameersbn-postgresql/tags and https://github.com/kkimurak/docker-postgresql

[mihelm]

@kkimurak Thanks a lot for posting that! With that I was able to upgrade, and the database migration went without a hitch.

[kkimurak]

@mihelm Glad it works.

---

For now, I plan to continue regular updates until the upstream repository is updated, I get tired of it, or I find a new place to migrate to. I might try linking it to an automatic dependency update service such as renovate, or setting up a github action to push periodically, while also learning about it. Of course, there are no guarantees.
Also, I haven't pushed to quay.io yet. I haven't signed up for the lowest development plan yet because it costs $15/month. I haven't decided whether to sign up or not.

[th-2021]

@kkimurak which tags should be used for your psql? I have 16.8-1-20250424, which works well, but I saw that you also pushed 16.8-1 without date.

[kkimurak]

@th-2021 Maybe :16 is suitable for most case. If you need fixed version tag you can specify {MAJOR-YYYYMMDD} (like 16-20250501) or image hash. I'm setting up to build all supported versions (currently 13.x to 17.x) every week so you can update by stop, rm, rmi and then pull new image.
I'm still thinking about what tag names format to use so the naming is differ (16-20250501) from last week (16.8-1-20250424)

Note that I'm using a script to build these images. This script will:

1. list up major versions of postgresql to support
2. update Dockerfile to set PG_VERSION to target major version and run docker build for each major versions
3. check output of psql --version
4. set tags ({MAJOR}, {MAJOR-YYYYMMDD}, {MAJOR.MINOR} and {MAJOR.MINOR-PATCH})

Finally I run docker push --all-tags. So, the latest version of postgresql that corresponds to the image tag name should be included. (That is, for example, if 16.9 is released in the future, tag:16 will include postgresql 16.9(-1), but tag:16.8 will remain as it was when 16.8 was installed at the time of build and will not be updated.)

[oboote]

I think this should probably be discussed in the sameersbn/docker-postgresql repository, not here (I don't know if there will be a response from the maintainers early on, i.e. before the release of gitlab 18 when postgresql16 becomes mandatory..)

Anyway, as a temporary workaround, I pushed kkimurak/sameersbn-postgresql I built myself. Currently :15 (:15.12, :15.12-1) and :16 (:16.8, :16.8-1) are available. I have confirmed that the scheduled backup using :16.8-1 works properly in combination with sameersbn/gitlab:17.11.0. See https://hub.docker.com/r/kkimurak/sameersbn-postgresql/tags

Is this a self-migrating postgresql image or does the db need to be migrated from 15-to-16 with a migrating-image before using this one?

[kkimurak]

@oboote Self-migrating (that runs pg_upgrade). Note that you need internet connection to install old version of postgresql.

The image kkimurak/sameersbn-postgresql (https://github.com/kkimurak/docker-postgresql) is forked from sameersbn/postgresql (https://github.com/sameersbn/docker-postgresql`) but the migration code is (at least for now) unchanged from upstream. See https://github.com/kkimurak/docker-postgresql/blob/5c15d58d2be8af61bc3ffb1b08a4fbd7ed2e505d/runtime/functions#L188-L234

[oboote]

Lovely - thanks 👍

[R-omk]

Small story:

First, I updated

following this scheme 17.1.8 => 17.3.7 => 17.5.5 => 17.8.7 => 17.11.3 (sameersbn/gitlab)

after each step, all background migrations were successfully completed

during this time, the image was sameersbn/postgresql:14-20230628

then, staying on version 17.11.3, I updated

sameersbn/postgresql:14-20230628 -> kkimurak/sameersbn-postgresql:16

however, gitlab refused to start

docker compose exec -it gitlab bash
tail -f /var/log/gitlab/supervisor/*.log

==> /var/log/gitlab/supervisor/sidekiq.log <==
undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module
/home/git/gitlab/lib/gitlab/current_settings.rb:37:in `method_missing'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:99:in `prometheus_metrics_enabled_unmemoized'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in `block in prometheus_metrics_enabled?'
/home/git/gitlab/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in `prometheus_metrics_enabled?'
/home/git/gitlab/config/initializers/7_prometheus_metrics.rb:59:in `<top (required)>'

---

==> /var/log/gitlab/supervisor/supervisord.log <==
2025-05-30 11:53:02,243 INFO exited: sidekiq (exit status 1; not expected)
2025-05-30 11:53:02,543 INFO spawned: 'sidekiq' with pid 3086
2025-05-30 11:53:03,582 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

---

==> /var/log/gitlab/supervisor/puma.log <==
{"timestamp":"2025-05-30T08:53:12.652Z","pid":3080,"message":"! Unable to load application: NoMethodError: undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module"}
bundler: failed to load command: puma (/home/git/gitlab/vendor/bundle/ruby/3.2.0/bin/puma)
/home/git/gitlab/lib/gitlab/current_settings.rb:37:in `method_missing': undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module (NoMethodError)
	from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:99:in `prometheus_metrics_enabled_unmemoized'
	from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in `block in prometheus_metrics_enabled?'
	from /home/git/gitlab/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
	from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in `prometheus_metrics_enabled?'
	from /home/git/gitlab/config/initializers/7_prometheus_metrics.rb:59:in `<top (required)>'

processes puma and sidekiq were stuck in a loop, restarting

---

I connected to the database

psql -U gitlab gitlabhq_production

\c gitlabhq_production

\dt

and discovered that there were only 59 tables in the database, I realized that the update had failed, I rolled back, gitlab started working,

in the database there were 884 tables

for now, I'm stopping attempts to upgrade pg 14->16

[th-2021]

During the pg update you should check the log. It should show you the migration process. Then wait until migration has finished. This is how it worked for me. Am 30. Mai 2025 11:45:08 MESZ schrieb R-omk ***@***.***>:

…

R-omk left a comment (sameersbn/docker-gitlab#3096) ### Small story: First, I updated following this scheme 17.1.8 => 17.3.7 => 17.5.5 => 17.8.7 => 17.11.3 (sameersbn/gitlab) after each step, all background migrations were successfully completed during this time, the image was sameersbn/postgresql:14-20230628 then, staying on version 17.11.3, I updated sameersbn/postgresql:14-20230628 -> kkimurak/sameersbn-postgresql:16 however, gitlab refused to start ```bash docker compose exec -it gitlab bash tail -f /var/log/gitlab/supervisor/*.log ``` ```log ==> /var/log/gitlab/supervisor/sidekiq.log <== undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module /home/git/gitlab/lib/gitlab/current_settings.rb:37:in `method_missing' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:99:in `prometheus_metrics_enabled_unmemoized' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in `block in prometheus_metrics_enabled?' /home/git/gitlab/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in `prometheus_metrics_enabled?' /home/git/gitlab/config/initializers/7_prometheus_metrics.rb:59:in `<top (required)>' --- ==> /var/log/gitlab/supervisor/supervisord.log <== 2025-05-30 11:53:02,243 INFO exited: sidekiq (exit status 1; not expected) 2025-05-30 11:53:02,543 INFO spawned: 'sidekiq' with pid 3086 2025-05-30 11:53:03,582 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) --- ==> /var/log/gitlab/supervisor/puma.log <== {"timestamp":"2025-05-30T08:53:12.652Z","pid":3080,"message":"! Unable to load application: NoMethodError: undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module"} bundler: failed to load command: puma (/home/git/gitlab/vendor/bundle/ruby/3.2.0/bin/puma) /home/git/gitlab/lib/gitlab/current_settings.rb:37:in `method_missing': undefined method `prometheus_metrics_enabled' for Gitlab::CurrentSettings:Module (NoMethodError) from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:99:in `prometheus_metrics_enabled_unmemoized' from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in `block in prometheus_metrics_enabled?' from /home/git/gitlab/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize' from /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in `prometheus_metrics_enabled?' from /home/git/gitlab/config/initializers/7_prometheus_metrics.rb:59:in `<top (required)>' ``` processes puma and sidekiq were stuck in a loop, restarting --- I connected to the database psql -U gitlab gitlabhq_production \c gitlabhq_production \dt and discovered that there were only 59 tables in the database, I realized that the update had failed, I rolled back, gitlab started working, in the database there were 884 tables for now, I'm stopping attempts to upgrade pg 14->16 -- Reply to this email directly or view it on GitHub: #3096 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[ArdeSeadis]

Small story:

First, I updated

following this scheme 17.1.8 => 17.3.7 => 17.5.5 => 17.8.7 => 17.11.3 (sameersbn/gitlab)

after each step, all background migrations were successfully completed

during this time, the image was sameersbn/postgresql:14-20230628

for now, I'm stopping attempts to upgrade pg 14->16

You should upgrade before to PG 14> to PG15 ! (so sameersbn/postgresql:15) and then kkimurak
not PG 14 to PG16

exemple for me :

‣ Migrating PostgreSQL 15 data to 16...
‣ Installing PostgreSQL 15...
(...)
‣ Migration in progress. Please be patient...Performing Consistency Checks
-----------------------------
Checking cluster versions                                     ok
Checking database user is the install user                    ok