Skip to content

Commit 71868da

Browse files
razvanxeniapesbernauerlfrancke
authored
feat(hbase): version 2.6.0 (#701)
* feat(hbase): version 2.6.0 * re-enable phoenix for 2.6 * fix lint and revert shell format * chore: change loglevel input format for opa process-logs script to uppercase (#704) * change loglevel input format to uppercase * update changelog * Revert "fix lint and revert shell format" This reverts commit 24249c5. * hbase-operator-tools for 2.6 * fix markdown lint * fix markdown lint * add xml dependency * package opa authorizer * fix hadolint warnings * update changelog * cleanup * Fix build * more indentation * update phoenix src archive * - convert tabs to spaces - add HBASE-28567 path * Added patches for open issues * conditionally bundle the jmx exporter * HBase 2.4.18 support - Patches updated for new version - Update dependencies for 2.4.18 to new patch dependencies where available - Move to heredoc syntax - Remove Log4Shell workarounds - Add package_manifest - Change upload script to only download & upload the sources and not the binaries * Changelog update * Hopefully fix path issues * Fix build and make build less noisy * Update labels * Fix typo * wip: delete_caches and Dockerfile refactoring * attempt: trigger image build from pr * Revert "attempt: trigger image build from pr" This reverts commit 9378bdf. * Updates - Move Phoenix into its own stage - Add ARG to delete caches - Use cache mounts * Fix changelog, remove hadolint warnings * Separate the hadoop-builder from hbase-builder so they can build in parallel * Split HBase Operator Tools into own stage as well, fix errors wi * fix missing copy * fix jmx exporter * fix jmx config * add jackson_dataformat_xml variable * add opa_authorizer variable * add opa-authorizer build stage * Updates - Move patches to stgit format - Include dataformat-xml in build itself * readd opa authorizer version * Use tag for the opa authorizer * implement review feedback * Move to identifiable Phoenix & HBase Operator tool dependencies * Fix markdownlint * Restore missing comment * Fix the build The snapshot builds needed for HBase 2.6 caused issues as the directory names for the source wouldn't correspond to the target directory that is being built * Silence hadolint * Silence hadolint --------- Co-authored-by: Xenia <xenia.fischer@stackable.tech> Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.tech> Co-authored-by: Lars Francke <git@lars-francke.de>
1 parent fc532b2 commit 71868da

13 files changed

Lines changed: 714 additions & 11 deletions

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ All notable changes to this project will be documented in this file.
2424
- hadoop: Add version `3.4.0` ([#743]).
2525
- druid: Build from source ([#684], [#696]).
2626
- opa: Add log processing script to opa for decision logging ([#695], [#704]).
27+
- hbase: Added new image for 2.6.0 including the new OPA authorizer ([#701]).
2728
- stackable-base: Add [config-utils](https://github.com/stackabletech/config-utils) ([#706]).
2829
- omid: Include Apache Omid Examples to simplify testing ([#721]).
2930
- nifi: Add support for 1.27.0 and 2.0.0-M4 ([#744], [#767]).
@@ -112,6 +113,7 @@ All notable changes to this project will be documented in this file.
112113
[#688]: https://github.com/stackabletech/docker-images/pull/688
113114
[#696]: https://github.com/stackabletech/docker-images/pull/696
114115
[#695]: https://github.com/stackabletech/docker-images/pull/695
116+
[#701]: https://github.com/stackabletech/docker-images/pull/701
115117
[#703]: https://github.com/stackabletech/docker-images/pull/703
116118
[#704]: https://github.com/stackabletech/docker-images/pull/704
117119
[#706]: https://github.com/stackabletech/docker-images/pull/706

‎hbase/Dockerfile‎

Lines changed: 55 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ USER stackable
3131
WORKDIR /stackable
3232

3333
COPY --chown=stackable:stackable hbase/stackable/patches /stackable/patches
34-
COPY --chown=stackable:stackable hbase/stackable/jmx /stackable/jmx
34+
COPY --chown=stackable:stackable hbase/stackable/jmx/config${JMX_EXPORTER} /stackable/jmx
3535

3636
# Cache mounts are owned by root by default
3737
# We need to explicitly give the uid to use which is hardcoded to "1000" in stackable-base
@@ -40,7 +40,7 @@ COPY --chown=stackable:stackable hbase/stackable/jmx /stackable/jmx
4040
# While this might work in theory it didn't in practice (FileNotFound exceptions etc.)
4141

4242
# Ignoring this because we set the shell to bash in our base image and hadolint/shellcheck doesn't know about it
43-
# hadolint ignore=SC3060
43+
# hadolint ignore=SC3060,SC3010
4444
RUN --mount=type=cache,id=maven-hbase,uid=1000,target=/stackable/.m2/repository <<EOF
4545
###
4646
### HBase
@@ -64,9 +64,11 @@ ln -s "/stackable/hbase-${PRODUCT}" /stackable/hbase
6464
###
6565
### JMX Prometheus Exporter/Agent
6666
###
67-
curl --fail -L "https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" -o "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar"
68-
chmod +x "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar"
69-
ln -s "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" /stackable/jmx/jmx_prometheus_javaagent.jar
67+
if [[ -n "${JMX_EXPORTER}" ]] ; then
68+
curl --fail -L "https://repo.stackable.tech/repository/packages/jmx-exporter/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" -o "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar"
69+
chmod +x "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar"
70+
ln -s "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" /stackable/jmx/jmx_prometheus_javaagent.jar
71+
fi
7072

7173
###
7274
### Async Profiler
@@ -87,6 +89,41 @@ if [ "${DELETE_CACHES}" = "true" ] ; then
8789
fi
8890
EOF
8991

92+
# hadolint ignore=DL3006
93+
FROM stackable/image/java-devel AS opa-authorizer-builder
94+
95+
ARG OPA_AUTHORIZER
96+
ARG DELETE_CACHES
97+
98+
USER stackable
99+
WORKDIR /stackable
100+
101+
# Ignoring this because we set the shell to bash in our base image and hadolint/shellcheck doesn't know about it
102+
# hadolint ignore=SC3060,SC3010
103+
RUN --mount=type=cache,id=maven-opa,uid=1000,target=/stackable/.m2/repository <<EOF
104+
105+
###
106+
### OPA Authorizer (only for 2.6 upwards)
107+
###
108+
if [[ -n "$OPA_AUTHORIZER" ]]; then
109+
git clone --depth 1 --branch "$OPA_AUTHORIZER" https://github.com/stackabletech/hbase-opa-authorizer.git
110+
mvn \
111+
--batch-mode \
112+
--no-transfer-progress \
113+
package \
114+
-DskipTests \
115+
-fhbase-opa-authorizer
116+
else
117+
# Create a dummy jar to avoid errors when copying it the final image
118+
mkdir -p hbase-opa-authorizer/target
119+
touch hbase-opa-authorizer/target/hbase-opa-authorizer.jar
120+
fi
121+
122+
123+
if [ "${DELETE_CACHES}" = "true" ] ; then
124+
rm -rf /stackable/.m2/repository/*
125+
fi
126+
EOF
90127

91128
# hadolint ignore=DL3006
92129
FROM stackable/image/java-devel AS hbase-operator-tools-builder
@@ -123,9 +160,13 @@ mvn \
123160
-fhbase-operator-tools-${HBASE_OPERATOR_TOOLS}-src \
124161
package assembly:single
125162

163+
# We need the "*" here as the directory won't be the same as the final tar file for SNAPSHOTs which we currently have to use for 2.6
164+
# And we're stripping the top level directory while extracting because it may be called different than the folder name when it's a SNAPSHOT
165+
mkdir /stackable/hbase-operator-tools-${HBASE_OPERATOR_TOOLS}
126166
tar -xz \
127-
-f hbase-operator-tools-${HBASE_OPERATOR_TOOLS}-src/hbase-operator-tools-assembly/target/hbase-operator-tools-${HBASE_OPERATOR_TOOLS}-bin.tar.gz \
128-
-C /stackable/
167+
-f hbase-operator-tools-${HBASE_OPERATOR_TOOLS}-src/hbase-operator-tools-assembly/target/hbase-operator-tools-*-bin.tar.gz \
168+
-C /stackable/hbase-operator-tools-${HBASE_OPERATOR_TOOLS}/ \
169+
--strip-components=1
129170
rm -rf /stackable/hbase-operator-tools-${HBASE_OPERATOR_TOOLS}-src
130171

131172
envsubst '${PRODUCT}:${HBASE_OPERATOR_TOOLS}' < /stackable/bin/hbck2.env > /stackable/bin/hbck2
@@ -210,9 +251,12 @@ mvn \
210251
clean \
211252
package
212253

213-
tar -xz -f phoenix-${PHOENIX}-src/phoenix-assembly/target/phoenix-hbase-${HBASE_PROFILE}-${PHOENIX}-bin.tar.gz -C /stackable/
254+
# We need the "*" here as the directory won't be the same as the final tar file for SNAPSHOTs which we currently have to use for 2.6
255+
# And we're stripping the top level directory while extracting because it may be called different than the folder name when it's a SNAPSHOT
256+
mkdir /stackable/phoenix-${HBASE_PROFILE}-${PHOENIX}-bin
257+
tar -xz -f phoenix-${PHOENIX}-src/phoenix-assembly/target/phoenix-hbase-*-bin.tar.gz -C /stackable/phoenix-${HBASE_PROFILE}-${PHOENIX}-bin/ --strip-components=1
214258
rm -rf /stackable/phoenix-${PHOENIX}-src
215-
ln -s "/stackable/phoenix-hbase-${HBASE_PROFILE}-${PHOENIX}-bin" /stackable/phoenix
259+
ln -s "/stackable/phoenix-${HBASE_PROFILE}-${PHOENIX}-bin" /stackable/phoenix
216260

217261

218262
# We're removing these to make the intermediate layer smaller
@@ -271,6 +315,8 @@ COPY --chown=stackable:stackable --from=phoenix-builder /stackable/phoenix /stac
271315
COPY --chown=stackable:stackable --from=hadoop-s3-builder /stackable/bin/export-snapshot-to-s3 /stackable/bin/export-snapshot-to-s3
272316
COPY --chown=stackable:stackable --from=hadoop-s3-builder /stackable/hadoop/share/hadoop/tools/lib/ /stackable/hadoop/share/hadoop/tools/lib/
273317

318+
COPY --chown=stackable:stackable --from=opa-authorizer-builder /stackable/hbase-opa-authorizer/target/hbase-opa-authorizer*.jar /stackable/hbase-${PRODUCT}/lib
319+
274320
RUN <<EOF
275321
microdnf update
276322

‎hbase/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Support for HBase 2.6
2+
3+
As of SDP 24.7 we do include HBase 2.6 support in an experimental state.
4+
5+
## Phoenix
6+
7+
At the time of this writing (July 12, 2024) there is no Phoenix release that supports HBase 2.6 so the script `upload_new_phoenix_version.sh` will not work.
8+
9+
HBase 2.6 support [was added](https://github.com/apache/phoenix/pull/1793) with [PHOENIX-7172](https://issues.apache.org/jira/browse/PHOENIX-7172).
10+
SDP 24.7 includes Phoenix built from the master branch from commit [4afe457](https://github.com/apache/phoenix/tree/4afe4579bb3ab01725e4939746d0b7b807b438ac).
11+
12+
```bash
13+
# clone the Phoenix repo
14+
git clone git@github.com:apache/phoenix.git
15+
cd phoenix
16+
git checkout 4afe457
17+
18+
# create a tarball
19+
mkdir ../phoenix-5.3.0-4afe457
20+
git archive --format=tar --output ../phoenix-5.3.0-4afe457/phoenix.tar 4afe457
21+
cd ../phoenix-5.3.0-4afe457
22+
tar xf phoenix.tar
23+
rm phoenix.tar
24+
echo 4afe457 > git-commit
25+
cd ..
26+
tar -c phoenix-5.3.0-4afe457 | gzip > phoenix-5.3.0-4afe457-src.tar.gz
27+
```
28+
29+
## HBase operator tools
30+
31+
Repository: [hbase-operator-tools](https://github.com/apache/hbase-operator-tools)
32+
33+
Built from git hash [7c738fc](https://github.com/apache/hbase-operator-tools/tree/7c738fc1bd14fd3e2ca4e66569b496b3fd9d0288) (master)
34+
since no release supporting HBase 2.6 available yet.
35+
36+
```bash
37+
mkdir ../hbase-operator-tools-1.3.0-7c738fc
38+
git archive --format=tar --output ../hbase-operator-tools-1.3.0-7c738fc/hot.tar 7c738fc
39+
cd ../hbase-operator-tools-1.3.0-7c738fc
40+
tar xf hot.tar
41+
rm hot.tar
42+
echo 7c738fc > git-commit
43+
cd ..
44+
tar -c hbase-operator-tools-1.3.0-7c738fc|gzip > hbase-operator-tools-1.3.0-7c738fc-src.tar.gz
45+
```

‎hbase/stackable/jmx/config/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Description
2+
3+
This is a placeholder file for docker to be able to build images when there is no jmx config provided.
4+
5+
This is the case for HBase 2.6.0 (and above) where a JMX exporter is included in the HBase distribution.

hbase/stackable/jmx/master.yaml renamed to hbase/stackable/jmx/config1.0.1/master.yaml

File renamed without changes.

hbase/stackable/jmx/regionserver.yaml renamed to hbase/stackable/jmx/config1.0.1/regionserver.yaml

File renamed without changes.

hbase/stackable/jmx/restserver.yaml renamed to hbase/stackable/jmx/config1.0.1/restserver.yaml

File renamed without changes.

0 commit comments

Comments
 (0)