Movable DataBase Locales for Cloudberry (#1363)

* Movable DataBase Locales for Cloudberry

We inherited this issue from PostgreSQL.

PostgreSQL uses glibc to sort strings. In version glibc=2.28, collations
broke down badly (in general, there are no guarantees when updating glibc).
Changing collations breaks indexes. Similarly, a cluster with different
collations also behaves unpredictably.

What and when something has changed in glibc can be found
on https://github.com/ardentperf/glibc-unicode-sorting
Also there is special postgresql-wiki https://wiki.postgresql.org/wiki/Locale_data_changes
And you tube video https://www.youtube.com/watch?v=0E6O-V8Jato

In short, the issue can be seen through the use of bash:

( echo "1-1"; echo "11" ) | LC_COLLATE=en_US.UTF-8 sort

gives the different results in ubunru 18.04 and 22.04.

There is no way to solve the problem other than by not changing the symbol order.
We freeze symbol order and use it instead of glibc.

Here the solution https://github.com/postgredients/mdb-locales.

In this PR I have added PostgreSQL patch that replaces all glibc
locale-related calls with a calls to an external libary. It activates
using new configure parameter --with-mdblocales, which is off by
default.

Using custom locales needs libmdblocales1 package and mdb-locales
package with symbol table.

Build needs libmdblocales-dev package with headers.

Fixing the symbol order is necessary for OS upgrade. For example Ubuntu 22.04 EOL is April 2027, Rocky 8 Active Support ended May 2024, and Security support ends in 2029.

We use Movable DataBase Locales in Greenplum 6 and all our PostgreSQL installations (starting with PostgreSQL 12). This patch is adopted patch version from our internal PostgreSQL 14 fork.

* mdb_admin role

This patch introcudes new pseudo-pre-defined role "mdb_admin".

Introduces 2 new function:
extern bool mdb_admin_allow_bypass_owner_checks(Oid userId,  Oid ownerId);
extern void check_mdb_admin_is_member_of_role(Oid member, Oid role);

To check mdb admin belongship and role-to-role ownership transfer
correctness.

Our mdb_admin ACL model is the following:
 * Any roles user or/and roles can be granted with mdb_admin
 * mdb_admin memeber can tranfser ownershup of relations,
   namespaces and functions to other roles, if target role in neither:
   superuser, pg_read_server_files, pg_write_server_files nor
   pg_execute_server_program.

* mdb_superuser role

This patch introcudes new pseudo-pre-defined role "mdb_superuser".

Role is capable of:

GRANT/REVOKE any set of priviledges to/from any object in database.
Has power of pg_database_owner in any database, including:
DROP any object in database (except system catalog and stuff)

Role is NOT capable of:

Create database, role, extension or alter other roles with such
priviledges.

Transfer ownership to /pass has_priv of roles:

PG_READ_ALL_DATA
PG_WRITE_ALL_DATA
PG_EXECUTE_SERVER_PROGRAM
PG_READ_SERVER_FILES
PG_WRITE_SERVER_FILES
PG_DATABASE_OWNER

Allow mdb_superuser to alter objects and grant ACl to
objects, owner by pg_database_owner. Also, when acl check,
allow mdb_supersuer use pg_database_owner role power to pass check

* Extend multixact SLRU

The issue here is the same as for the PG, good detail description I found in Nikolay blog post https://v2.postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful  
See also the history of original PG patches in https://commitfest.postgresql.org/patch/2627/  We could get all those fixes after rebasing to PG18, but for now, we need to adjust SLRU structure sizes.

---------
Co-authored-by: usernamedt <usernamedt@yandex-team.com>
Co-authored-by: reshke <reshkekirill@gmail.com>

Author: 3 people

configure

@@ -698,6 +698,7 @@ BISON
 MKDIR_P
 LN_S
 TAR
+USE_MDBLOCALES
 install_bin
 INSTALL_DATA
 INSTALL_SCRIPT
@@ -945,6 +946,7 @@ with_rt
 with_libcurl
 with_apr_config
 with_gnu_ld
+with_mdblocales
 with_ssl
 with_openssl
 enable_openssl_redirect
@@ -1693,6 +1695,7 @@ Optional Packages:
   --without-libcurl       do not use libcurl
   --with-apr-config=PATH  path to apr-1-config utility
   --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
+  --without-mdblocales    build without MDB locales
   --with-ssl=LIB          use LIB for SSL/TLS support (openssl)
   --with-openssl          obsolete spelling of --with-ssl=openssl
 
@@ -2909,7 +2912,6 @@ PG_PACKAGE_VERSION=14.4
 
 
 
-
 ac_aux_dir=
 for ac_dir in config "$srcdir"/config; do
   if test -f "$ac_dir/install-sh"; then
@@ -12208,6 +12210,38 @@ case $INSTALL in
 esac
 
 
+#
+# MDB locales
+#
+
+
+
+
+# Check whether --with-mdblocales was given.
+if test "${with_mdblocales+set}" = set; then :
+  withval=$with_mdblocales;
+  case $withval in
+    yes)
+
+$as_echo "#define USE_MDBLOCALES 1" >>confdefs.h
+
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-mdblocales option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_mdblocales=no
+
+fi
+
+
+
+
 if test -z "$TAR"; then
   for ac_prog in tar
 do
@@ -12844,6 +12878,56 @@ $as_echo "${python_libspec} ${python_additional_libs}" >&6; }
 
 
 
+fi
+
+if test "$with_mdblocales" = yes; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for mdb_setlocale in -lmdblocales" >&5
+$as_echo_n "checking for mdb_setlocale in -lmdblocales... " >&6; }
+if ${ac_cv_lib_mdblocales_mdb_setlocale+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lmdblocales  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char mdb_setlocale ();
+int
+main ()
+{
+return mdb_setlocale ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_mdblocales_mdb_setlocale=yes
+else
+  ac_cv_lib_mdblocales_mdb_setlocale=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_mdblocales_mdb_setlocale" >&5
+$as_echo "$ac_cv_lib_mdblocales_mdb_setlocale" >&6; }
+if test "x$ac_cv_lib_mdblocales_mdb_setlocale" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBMDBLOCALES 1
+_ACEOF
+
+  LIBS="-lmdblocales $LIBS"
+
+else
+  as_fn_error $? "mdblocales library not found" "$LINENO" 5
+fi
+
 fi
 
 if test x"$cross_compiling" = x"yes" && test -z "$with_system_tzdata"; then
@@ -17065,6 +17149,17 @@ fi
 
 done
 
+fi
+
+if test "$with_mdblocales" = yes; then
+  ac_fn_c_check_header_mongrel "$LINENO" "mdblocales.h" "ac_cv_header_mdblocales_h" "$ac_includes_default"
+if test "x$ac_cv_header_mdblocales_h" = xyes; then :
+
+else
+  as_fn_error $? "mdblocales header not found." "$LINENO" 5
+fi
+
+
 fi
 
 if test "$with_gssapi" = yes ; then

configure.ac

@@ -1462,6 +1462,14 @@ case $INSTALL in
 esac
 AC_SUBST(install_bin)
 
+#
+# MDB locales
+#
+
+PGAC_ARG_BOOL(with, mdblocales, yes, [build without MDB locales],
+              [AC_DEFINE([USE_MDBLOCALES], 1, [Define to 1 to build with MDB locales. (--with-mdblocales)])])
+AC_SUBST(USE_MDBLOCALES)
+
 PGAC_PATH_PROGS(TAR, tar)
 AC_PROG_LN_S
 AC_PROG_MKDIR_P
@@ -1620,6 +1628,11 @@ failure.  It is possible the compiler isn't looking in the proper directory.
 Use --without-zlib to disable zlib support.])])
 fi
 
+if test "$with_mdblocales" = yes; then
+  AC_CHECK_LIB(mdblocales, mdb_setlocale, [],
+               [AC_MSG_ERROR([mdblocales library not found])])
+fi
+
 if test "$enable_external_fts" = yes; then
 AC_CHECK_LIB(jansson, jansson_version_str, [],
                [AC_MSG_ERROR([jansson library not found or version is too old, version must >= 2.13])])
@@ -1999,6 +2012,10 @@ if test "$with_lz4" = yes; then
   AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
 fi
 
+if test "$with_mdblocales" = yes; then
+  AC_CHECK_HEADER(mdblocales.h, [], [AC_MSG_ERROR([mdblocales header not found.])])
+fi
+
 if test "$with_gssapi" = yes ; then
   AC_CHECK_HEADERS(gssapi/gssapi.h, [],
 	[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])

contrib/pax_storage/src/cpp/storage/oper/pax_oper.cc

@@ -25,6 +25,7 @@
  *-------------------------------------------------------------------------
  */
 
+#include "common/mdb_locale.h"
 #include "storage/oper/pax_oper.h"
 
 #include "comm/cbdb_wrappers.h"
@@ -588,9 +589,9 @@ static inline bool LocaleIsC(Oid collation) {
       return (bool)result;
     }
 
-    localeptr = setlocale(LC_COLLATE, NULL);
+    localeptr = SETLOCALE(LC_COLLATE, NULL);
     CBDB_CHECK(localeptr, cbdb::CException::ExType::kExTypeCError,
-               fmt("Invalid locale, fail to `setlocale`, errno: %d", errno));
+               fmt("Invalid locale, fail to `SETLOCALE`, errno: %d", errno));
 
     if (strcmp(localeptr, "C") == 0 ||  // cut line
         strcmp(localeptr, "POSIX") == 0) {

contrib/pax_storage/src/test/regress/expected/create_function_3.out

@@ -166,10 +166,10 @@ SET SESSION AUTHORIZATION regress_unpriv_user;
 SET search_path TO temp_func_test, public;
 ALTER FUNCTION functest_E_1(int) NOT LEAKPROOF;
 ALTER FUNCTION functest_E_2(int) LEAKPROOF;
-ERROR:  only superuser can define a leakproof function
+ERROR:  only superuser or mdb_admin can define a leakproof function
 CREATE FUNCTION functest_E_3(int) RETURNS bool LANGUAGE 'sql'
        LEAKPROOF AS 'SELECT $1 < 200';	-- fail
-ERROR:  only superuser can define a leakproof function
+ERROR:  only superuser or mdb_admin can define a leakproof function
 RESET SESSION AUTHORIZATION;
 --
 -- CALLED ON NULL INPUT | RETURNS NULL ON NULL INPUT | STRICT

contrib/pax_storage/src/test/regress/expected/create_function_3_optimizer.out

@@ -166,10 +166,10 @@ SET SESSION AUTHORIZATION regress_unpriv_user;
 SET search_path TO temp_func_test, public;
 ALTER FUNCTION functest_E_1(int) NOT LEAKPROOF;
 ALTER FUNCTION functest_E_2(int) LEAKPROOF;
-ERROR:  only superuser can define a leakproof function
+ERROR:  only superuser or mdb_admin can define a leakproof function
 CREATE FUNCTION functest_E_3(int) RETURNS bool LANGUAGE 'sql'
        LEAKPROOF AS 'SELECT $1 < 200';	-- fail
-ERROR:  only superuser can define a leakproof function
+ERROR:  only superuser or mdb_admin can define a leakproof function
 RESET SESSION AUTHORIZATION;
 --
 -- CALLED ON NULL INPUT | RETURNS NULL ON NULL INPUT | STRICT

devops/build/automation/cloudberry/scripts/configure-cloudberry.sh

@@ -62,6 +62,12 @@
 #                   --enable-cassert
 #                   --enable-debug-extensions
 #
+#   ENABLE_MDBLOCALES - Enable custom locales (true/false, defaults to
+#                       false)
+#
+#                 When true, add option:
+#                   --with-mdblocales
+#
 # Prerequisites:
 #   - System dependencies must be installed:
 #     * xerces-c development files
@@ -138,6 +144,11 @@ if [ "${ENABLE_DEBUG:-false}" = "true" ]; then
                           --enable-debug-extensions"
 fi
 
+CONFIGURE_MDBLOCALES_OPTS="--without-mdblocales"
+if [ "${ENABLE_MDBLOCALES:-false}" = "true" ]; then
+    CONFIGURE_MDBLOCALES_OPTS="--with-mdblocales"
+fi
+
 # Configure build
 log_section "Configure"
 execute_cmd ./configure --prefix=${BUILD_DESTINATION} \
@@ -164,6 +175,7 @@ execute_cmd ./configure --prefix=${BUILD_DESTINATION} \
             --with-ssl=openssl \
             --with-openssl \
             --with-uuid=e2fs \
+            ${CONFIGURE_MDBLOCALES_OPTS} \
             --with-includes=/usr/local/xerces-c/include \
             --with-libraries=${BUILD_DESTINATION}/lib || exit 4
 log_section_end "Configure"

gpcontrib/orafce/others.c

@@ -45,6 +45,7 @@
 #include "utils/uuid.h"
 #include "orafce.h"
 #include "builtins.h"
+#include "common/mdb_locale.h"
 
 /*
  * Source code for nlssort is taken from postgresql-nls-string
@@ -322,7 +323,7 @@ _nls_run_strxfrm(text *string, text *locale)
 	 */
 	if (!lc_collate_cache)
 	{
-		if ((lc_collate_cache = setlocale(LC_COLLATE, NULL)))
+		if ((lc_collate_cache = SETLOCALE(LC_COLLATE, NULL)))
 			/* Make a copy of the locale name string. */
 #ifdef _MSC_VER
 			lc_collate_cache = _strdup(lc_collate_cache);
@@ -364,7 +365,7 @@ _nls_run_strxfrm(text *string, text *locale)
 			 * If setlocale failed, we know the default stayed the same,
 			 * co we can safely elog.
 			 */
-			if (!setlocale(LC_COLLATE, locale_str))
+			if (!SETLOCALE(LC_COLLATE, locale_str))
 				elog(ERROR, "failed to set the requested LC_COLLATE value [%s]", locale_str);
 
 			changed_locale = true;
@@ -409,7 +410,7 @@ _nls_run_strxfrm(text *string, text *locale)
 			/*
 			 * Set original locale
 			 */
-			if (!setlocale(LC_COLLATE, lc_collate_cache))
+			if (!SETLOCALE(LC_COLLATE, lc_collate_cache))
 				elog(FATAL, "failed to set back the default LC_COLLATE value [%s]", lc_collate_cache);
 		}
 
@@ -422,7 +423,7 @@ _nls_run_strxfrm(text *string, text *locale)
 		/*
 		 * Set original locale
 		 */
-		if (!setlocale(LC_COLLATE, lc_collate_cache))
+		if (!SETLOCALE(LC_COLLATE, lc_collate_cache))
 			elog(FATAL, "failed to set back the default LC_COLLATE value [%s]", lc_collate_cache);
 		pfree(locale_str);
 	}

src/backend/catalog/namespace.c

@@ -2971,7 +2971,6 @@ LookupExplicitNamespace(const char *nspname, bool missing_ok)
 {
 	Oid			namespaceId;
 	AclResult	aclresult;
-
 	/* check for pg_temp alias */
 	if (strcmp(nspname, "pg_temp") == 0)
 	{
@@ -2989,7 +2988,24 @@ LookupExplicitNamespace(const char *nspname, bool missing_ok)
 	if (missing_ok && !OidIsValid(namespaceId))
 		return InvalidOid;
 
-	aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_USAGE);
+	HeapTuple tuple;
+	Oid ownerId;
+
+	tuple = SearchSysCache1(NAMESPACEOID, ObjectIdGetDatum(namespaceId));
+	if (!HeapTupleIsValid(tuple))
+		ereport(ERROR,
+				(errcode(ERRCODE_UNDEFINED_SCHEMA),
+				 errmsg("schema with OID %u does not exist", namespaceId)));
+
+	ownerId = ((Form_pg_namespace) GETSTRUCT(tuple))->nspowner;
+
+	ReleaseSysCache(tuple);
+
+	if (!mdb_admin_allow_bypass_owner_checks(GetUserId(), ownerId)) {
+		aclresult = pg_namespace_aclcheck(namespaceId, GetUserId(), ACL_USAGE);
+	} else {
+		aclresult = ACLCHECK_OK;
+	}
 	if (aclresult != ACLCHECK_OK)
 		aclcheck_error(aclresult, OBJECT_SCHEMA,
 					   nspname);

src/backend/commands/alter.c

@@ -1085,7 +1085,8 @@ AlterObjectOwner_internal(Relation rel, Oid objectId, Oid new_ownerId)
 		if (!superuser())
 		{
 			/* must be owner */
-			if (!has_privs_of_role(GetUserId(), old_ownerId))
+			if (!has_privs_of_role(GetUserId(), old_ownerId) 
+			&& !mdb_admin_allow_bypass_owner_checks(GetUserId(), old_ownerId))
 			{
 				char	   *objname;
 				char		namebuf[NAMEDATALEN];
@@ -1105,14 +1106,13 @@ AlterObjectOwner_internal(Relation rel, Oid objectId, Oid new_ownerId)
 				aclcheck_error(ACLCHECK_NOT_OWNER, get_object_type(classId, objectId),
 							   objname);
 			}
-			/* Must be able to become new owner */
-			check_is_member_of_role(GetUserId(), new_ownerId);
+
+			check_mdb_admin_is_member_of_role(GetUserId(), new_ownerId);
 
 			/* New owner must have CREATE privilege on namespace */
 			if (OidIsValid(namespaceId))
 			{
 				AclResult	aclresult;
-
 				aclresult = pg_namespace_aclcheck(namespaceId, new_ownerId,
 												  ACL_CREATE);
 				if (aclresult != ACLCHECK_OK)