Strange xargs Behavior with special characters within variable

January 23, 2024

I’m running into strange behavior with xargs in a bash script where the replacement is not working anymore when placing it in the middle of a variable with special characters. Please see the following:

#!/bin/bash

url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"

echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}

results in:

https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 1
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 2
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 3

Please note the 1 2 and 3 did not show up but was the literal {} in the endpoint.

I determined it was the parameter because when I removed a few characters, it worked:

#!/bin/bash

url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"


echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}

https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3

Note the 1, 2, and 3 is there.

I am on MacOS. This behavior seems to be specific to the xargs on Mac. This will not even run on linux xargs with the flags passed. Would love some suggestions how to make this portable and work on all systems. Thank you.

>Solution :

From the man page:

-I replstr
        Execute utility for each input line, replacing one or more
        occurrences of replstr in up to replacements (or 5 if no -R flag
        is specified) arguments to utility with the entire line of input.
        The resulting arguments, after replacement is done, will not be
        allowed to grow beyond replsize (or 255 if no -S flag is
        specified) bytes; this is implemented by concatenating as much of
        the argument containing replstr as possible, to the constructed
        arguments to utility, up to replsize bytes.  The size limit does
        not apply to arguments to utility which do not contain replstr,
        and furthermore, no replacement will be done on utility itself.
        Implies -x.

Your output string is longer that 255 characters, so it doesn’t perform the replacement. Add a -S option that’s large than the maximum possible output string.

$ echo 1 2 3 | xargs -n1 -I{} -S 500 echo "$url{}$parameters" -o {}
https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3

Also remember to quote variables to prevent word splitting and globbing.