Skip to content

TOOLS-xxxx use more content-type: in manta uploads #1063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
danmcd wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

TOOLS-xxxx use more content-type: in manta uploads #1063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7e3ecbe
OS-xxxx use more content-type: in manta uploads
danmcd Nov 18, 2022
66c4e63
submodule
danmcd Nov 22, 2022
f2bf1e3
Fix .iso type
danmcd Nov 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deps/eng
Open in desktop
Submodule eng updated 1 files
+30 −1 tools/bits-upload.sh
33 changes: 32 additions & 1 deletion tools/build_ctftools
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

#
# Copyright 2020 Joyent, Inc.
# Copyright 2022 MNX Cloud, Inc.
#

#
Expand All @@ -23,6 +24,34 @@
usage="$0 make -j maxjobs -o tarball"
usage="$usage | upload -D bits -d path -p platform -t timestamp"

function content_type
{
filename="$1"
# basename <name> <.ext> will strip .ext off. Will be == name if it's
# not .ext.
if [[ "$(basename $filename .tgz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .gz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .txt)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .log)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .html)" != "$filename" ]]; then
content_type="content-type: text/html"
elif [[ "$(basename $filename .json)" != "$filename" ]]; then
content_type="content-type: application/json"
elif [[ "$(basename $filename .iso)" != "$filename" ]]; then
content_type="content-type: application/x-iso9660-image"
else
# Default value... XXX KEBE ASKS use text/plain instead?
content_type="content-type: application/octet-stream"
fi

# Caller should use quotes around this.
echo $content_type
}

function fatal
{
local msg="$*"
Expand Down Expand Up @@ -133,7 +162,9 @@ function upload
verbose mmkdir -p "$mbdir"

for file in $bits_dir/*; do
verbose mput -f "$file" "$mbdir/$(basename $file)"
filename="$(basename $file)"
verbose mput -H "$(content_type $filename)" -f "$file" \
Copy link
Copy Markdown
Contributor

@bahamat bahamat Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be:

-H "content-type: $(content_type $filename)"

Copy link
Copy Markdown
Contributor

@bahamat bahamat Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, wait...you're already including the header name. That should work. I guess we'll need to dig into Jenkins to see what went wrong.

Copy link
Copy Markdown
Contributor Author

@danmcd danmcd Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's TritonDataCenter/eng that needs some changes here too. Building one right now that I hope will include the content-type branch of eng.

Copy link
Copy Markdown
Contributor Author

@danmcd danmcd Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "upload-bits" script.

Copy link
Copy Markdown
Contributor Author

@danmcd danmcd Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See TritonDataCenter/eng#62

"$mbdir/$filename"
done

echo "$mbdir" | mput -H "content-type: text/plain" "$mdir/latest"
Expand Down
32 changes: 31 additions & 1 deletion tools/build_strap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,34 @@ max_jobs=$($wsroot/tools/optimize_jobs)
force_build="no"
tarfile=""

function content_type
{
filename="$1"
# basename <name> <.ext> will strip .ext off. Will be == name if it's
# not .ext.
if [[ "$(basename $filename .tgz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .gz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .txt)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .log)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .html)" != "$filename" ]]; then
content_type="content-type: text/html"
elif [[ "$(basename $filename .json)" != "$filename" ]]; then
content_type="content-type: application/json"
elif [[ "$(basename $filename .iso)" != "$filename" ]]; then
content_type="content-type: application/x-iso9660-image"
else
# Default value... XXX KEBE ASKS use text/plain instead?
content_type="content-type: application/octet-stream"
fi

# Caller should use quotes around this.
echo $content_type
}

function fatal
{
local msg="$*"
Expand Down Expand Up @@ -290,7 +318,9 @@ function upload_strap
verbose mmkdir -p "$mbdir"

for file in $bits_dir/*; do
verbose mput -f $file $mbdir/$(basename $file)
filename="$(basename $file)"
verbose mput -H "$(content_type $filename)" -f $file \
"$mbdir/$filename"
done

echo $mbdir | mput -H "content-type: text/plain" "$mdir/latest"
Expand Down
37 changes: 33 additions & 4 deletions tools/smartos-release
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

#
# Copyright 2020 Joyent, Inc.
# Copyright 2022 MNX Cloud, Inc.
#

#
Expand Down Expand Up @@ -41,6 +42,34 @@ fi

# --- support functions

function content_type
{
filename="$1"
# basename <name> <.ext> will strip .ext off. Will be == name if it's
# not .ext.
if [[ "$(basename $filename .tgz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .gz)" != "$filename" ]]; then
content_type="content-type: application/gzip"
elif [[ "$(basename $filename .txt)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .log)" != "$filename" ]]; then
content_type="content-type: text/plain"
elif [[ "$(basename $filename .html)" != "$filename" ]]; then
content_type="content-type: text/html"
elif [[ "$(basename $filename .json)" != "$filename" ]]; then
content_type="content-type: application/json"
elif [[ "$(basename $filename .iso)" != "$filename" ]]; then
content_type="content-type: application/x-iso9660-image"
else
# Default value... XXX KEBE ASKS use text/plain instead?
content_type="content-type: application/octet-stream"
fi

# Caller should use quotes around this.
echo $content_type
}

function fatal {
echo "$(basename $0): error: $1"
exit 1
Expand Down Expand Up @@ -98,14 +127,14 @@ BUILD_FILES="platform-${BRANCH}-${TIMESTAMP}.tgz
index.html
md5sums.txt"
for file in ${BUILD_FILES}; do
mput -f ${BITS}/$file ${SMARTOS_RELEASE}/$file
mput -H "$(content_type $file) -f ${BITS}/$file ${SMARTOS_RELEASE}/$file
done

echo "Uploading 'latest' ${SMARTOS} objects to ${SMARTOS}"
mput -f $BITS/platform-${BRANCH}-${TIMESTAMP}.tgz ${SMARTOS}/platform-latest.tgz
mput -H "content-type: application/gzip" -f $BITS/platform-${BRANCH}-${TIMESTAMP}.tgz ${SMARTOS}/platform-latest.tgz
mput -f $BITS/smartos-${TIMESTAMP}.iso ${SMARTOS}/smartos-latest.iso
mput -f $BITS/smartos-${TIMESTAMP}-USB.img.gz ${SMARTOS}/smartos-latest-USB.img.gz
mput -f $BITS/smartos-${TIMESTAMP}.vmwarevm.tar.gz ${SMARTOS}/smartos-latest.vmwarevm.tar.gz
mput -H "content-type: application/gzip" -f $BITS/smartos-${TIMESTAMP}-USB.img.gz ${SMARTOS}/smartos-latest-USB.img.gz
mput -H "content-type: application/gzip" -f $BITS/smartos-${TIMESTAMP}.vmwarevm.tar.gz ${SMARTOS}/smartos-latest.vmwarevm.tar.gz

echo "Updating ${SMARTOS}/latest object"
echo ${SMARTOS_RELEASE} | mput -H 'content-type: text/plain' ${SMARTOS}/latest
Expand Down