Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Remove last `},` in a dynamically generated JSON array using sed or jq

I am attempting to dynamically generate a JSON, however, the method I follow leaves a last }, making the JSON invalid:

#Initializing Current State file
    echo "[" > state.json
    # Getting list of all EKS clusters
    for cluster in $(aws eks list-clusters --output text | awk '{print $2}')
    do
      echo "====== $cluster ======"
      # Get list of all ASG from cluster
      for asg in $(aws autoscaling describe-auto-scaling-groups  --query 'AutoScalingGroups[?contains(Tags[?Key==`eks:cluster-name`].Value, `'"$cluster"'`)].[AutoScalingGroupName]' --output text)
      do
        echo "Getting capacity for ASG: $asg"
        MinSize=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].MinSize' --output text)
        DesiredCapacity=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].DesiredCapacity' --output text)  
        MaxSize=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].MaxSize' --output text)
        JSON_STRING=$( jq -n \
                  --arg cn "$cluster" \
                  --arg asg "$asg" \
                  --arg MinSize "$MinSize" \
                  --arg DesiredCapacity "$DesiredCapacity" \
                  --arg MaxSize "$MaxSize" \
                  '{cluster: $cn, asgname: $asg, minSize: $MinSize, DesiredCapacity: $DesiredCapacity, MaxSize: $MaxSize}' )
        echo "${JSON_STRING}," >> state.json
      done
    done
    #Finalizing Current State file
    echo "]" >> state.json
$ cat state.json
[
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "dev",
  "asgname": "eks-12345",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "dev",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-a",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-a",
  "asgname": "eks-1323",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
]

jq validation:

$ cat state.json  | jq empty
parse error: Expected another array element at line 44, column 1

I have attempted multiple options, mostly with sed, but have not met the specific requirement to only remove the last occurrence of },
Any advice is appreciated.

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

Let your script just produce a stream of objects by not printing the brackets and the commas.

echo "${JSON_STRING}" >> state.json

state.json should then contain a stream of objects:

{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "dev",
  "asgname": "eks-12345",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "dev",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-a",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-a",
  "asgname": "eks-1323",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}

Now you can read in that stream with jq, using the -s option to convert it into a proper array.

jq -s . state.json
[
  {
    "cluster": "eks-b",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-b",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "dev",
    "asgname": "eks-12345",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "dev",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-a",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-a",
    "asgname": "eks-1323",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  }
]

Demo for the jq -s part

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading