Download a file from Azure Blob Storage using Bash/Curl

Question

I'm trying to use the following script to download a file from Azure Blob Storage:

authorization="SharedKey"

HTTP_METHOD="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2009-09-19"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
x_ms_blob_type_h="x-ms-blob-type:BlockBlob"


# Build the signature string
canonicalized_headers="$${x_ms_date_h}\n$${x_ms_version_h}"
canonicalized_resource="/${STORAGE_ACCOUNT}/${STORAGE_CONTAINER}"

string_to_sign="$${HTTP_METHOD}\n\n\n\n\n\n\n\n\n\n\n\n$${x_ms_blob_type_h}\n$${canonicalized_headers}\n$${canonicalized_resource}"

# Decode the Base64 encoded access key, convert to Hex.

decoded_hex_key="$(echo -n ${STORAGE_KEY} | base64 -d -w0 | xxd -p -c256 | tr -d ' ')"

# Create the HMAC signature for the Authorization header
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)

authorization_header="Authorization: $authorization $STORAGE_ACCOUNT:$signature"
FILE_TYPE="application/x-yml"
DOWNLOAD_FILE="https://${STORAGE_ACCOUNT}.blob.core.windows.net/${STORAGE_CONTAINER}/${FILENAME}"

curl -H "$x_ms_date_h" \
     -H "$x_ms_version_h" \
     -H "$x_ms_blob_type_h" \
     -H "$authorization_header" \
     -H "Content-Type: $${FILE_TYPE}" \
     -f $${DOWNLOAD_FILE} -o ${FILENAME} 

I'm also using Terraform's template_file provider to call this script and I had to escape some of the variables, thus the weird interpolation. But I've debugged the script and all variables seem to be placed properly. The issue is somewhere with the SAS generation, as I keep getting this:

+ curl -H 'x-ms-date:Fri, 13 Sep 2019 11:04:40 GMT' -H x-ms-version:2009-09-19 -H x-ms-blob-type:BlockBlob -H 'Authorization: SharedKey *masked*:vyD7pp7Rqu3JBuS5IkHW0GMS2L82BN9fNKbmDAjuEoQ=' -H 'Content-
Type: application/octet-stream' -f https://*masked*.blob.core.windows.net/*masked*/*masked* -o *masked*
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Any ideas what I might be doing wrong here?

Answer 1

The problem is in your string_to_sign which is incomplete because variable canonicalized_resource is incomplete.

You should declare canonicalized_resource with complete URL like this :

canonicalized_resource="/${STORAGE_ACCOUNT}/${STORAGE_CONTAINER}/${FILENAME}"

Answer 2

I couldn't get this to work, so I ended up using terraform file providers to get the files on the node rather than pulling them from Azure Blob storage.