Extract values from a dict made of a list made of another dict

October 19, 2022

I have the following dict which is composed of a list

response = client.describe_auto_scaling_groups()

print(type(response),'\n')
for key,value in response.items():
    print(type(value))
    print(len(value))

the result to this is

<class 'dict'> 

<class 'list'>
2
<class 'dict'>
4

I am interested only in the list output and when I further dig into the type of data in the list, I get

    for key,value in response.items():
        print(type(value))
        print(len(value))
        for lst in value:
            print(type(lst))

I get

<class 'dict'> 

<class 'list'>
2
<class 'dict'>
<class 'dict'>
<class 'dict'>
4
<class 'str'>
<class 'str'>
<class 'str'>
<class 'str'>

So basically I have a 1 dict object, which has a 2 list object (corresponds to the 2 auto scaling groups) in it which has 4 str objects in it.

The response object looks like this

{'AutoScalingGroups': [{'AutoScalingGroupName': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'AutoScalingGroupARN': 'arn:aws:autoscaling:ca-central-1:744734775600:autoScalingGroup:e700fbf0-178e-4ff8-8687-6b7103af5b65:autoScalingGroupName/eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'MixedInstancesPolicy': {'LaunchTemplate': {'LaunchTemplateSpecification': {'LaunchTemplateId': 'lt-0477d3fffb684214d', 'LaunchTemplateName': 'eks-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'Version': '1'}, 'Overrides': [{'InstanceType': 't3.2xlarge'}]}, 'InstancesDistribution': {'OnDemandAllocationStrategy': 'prioritized', 'OnDemandBaseCapacity': 0, 'OnDemandPercentageAboveBaseCapacity': 0, 'SpotAllocationStrategy': 'capacity-optimized'}}, 'MinSize': 3, 'MaxSize': 6, 'DesiredCapacity': 3, 'DefaultCooldown': 300, 'AvailabilityZones': ['ca-central-1d', 'ca-central-1b', 'ca-central-1a'], 'LoadBalancerNames': [], 'TargetGroupARNs': ['arn:aws:elasticloadbalancing:ca-central-1:744734775600:targetgroup/eks-ingress-http/3dfa91118f700452', 'arn:aws:elasticloadbalancing:ca-central-1:744734775600:targetgroup/eks-ingress/e4a924db58da487a'], 'HealthCheckType': 'EC2', 'HealthCheckGracePeriod': 15, 'Instances': [{'InstanceId': 'i-020d70d503bc32dbc', 'InstanceType': 't3.2xlarge', 'AvailabilityZone': 'ca-central-1b', 'LifecycleState': 'InService', 'HealthStatus': 'Healthy', 'LaunchTemplate': {'LaunchTemplateId': 'lt-0477d3fffb684214d', 'LaunchTemplateName': 'eks-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'Version': '1'}, 'ProtectedFromScaleIn': False}, {'InstanceId': 'i-05da68bac7b47b5f6', 'InstanceType': 't3.2xlarge', 'AvailabilityZone': 'ca-central-1a', 'LifecycleState': 'InService', 'HealthStatus': 'Healthy', 'LaunchTemplate': {'LaunchTemplateId': 'lt-0477d3fffb684214d', 'LaunchTemplateName': 'eks-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'Version': '1'}, 'ProtectedFromScaleIn': False}, {'InstanceId': 'i-099e1d57a3cb55d93', 'InstanceType': 't3.2xlarge', 'AvailabilityZone': 'ca-central-1d', 'LifecycleState': 'InService', 'HealthStatus': 'Healthy', 'LaunchTemplate': {'LaunchTemplateId': 'lt-0477d3fffb684214d', 'LaunchTemplateName': 'eks-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'Version': '1'}, 'ProtectedFromScaleIn': False}], 'CreatedTime': datetime.datetime(2022, 10, 11, 20, 29, 50, 226000, tzinfo=tzutc()), 'SuspendedProcesses': [], 'VPCZoneIdentifier': 'subnet-0286d381da9f5fa9c,subnet-0fc58322f447bae71,subnet-0381004140a514844', 'EnabledMetrics': [{'Metric': 'GroupInServiceCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTotalInstances', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupAndWarmPoolDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupAndWarmPoolTotalCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolTerminatingCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolMinSize', 'Granularity': '1Minute'}, {'Metric': 'GroupMinSize', 'Granularity': '1Minute'}, {'Metric': 'GroupInServiceInstances', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolPendingCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolWarmedCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupStandbyCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTerminatingInstances', 'Granularity': '1Minute'}, {'Metric': 'GroupMaxSize', 'Granularity': '1Minute'}, {'Metric': 'GroupPendingInstances', 'Granularity': '1Minute'}, {'Metric': 'GroupStandbyInstances', 'Granularity': '1Minute'}, {'Metric': 'GroupTerminatingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupPendingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTotalCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolTotalCapacity', 'Granularity': '1Minute'}], 'Tags': [{'ResourceId': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'ResourceType': 'auto-scaling-group', 'Key': 'eks:cluster-name', 'Value': 'dev-build', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'ResourceType': 'auto-scaling-group', 'Key': 'eks:nodegroup-name', 'Value': 'core-20221011202902697400000006', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'ResourceType': 'auto-scaling-group', 'Key': 'k8s.io/cluster-autoscaler/dev-build', 'Value': 'owned', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'ResourceType': 'auto-scaling-group', 'Key': 'k8s.io/cluster-autoscaler/enabled', 'Value': 'true', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241', 'ResourceType': 'auto-scaling-group', 'Key': 'kubernetes.io/cluster/dev-build', 'Value': 'owned', 'PropagateAtLaunch': True}], 'TerminationPolicies': ['AllocationStrategy', 'OldestLaunchTemplate', 'OldestInstance'], 'NewInstancesProtectedFromScaleIn': False, 'ServiceLinkedRoleARN': 'arn:aws:iam::744734775600:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling', 'CapacityRebalance': True}, {'AutoScalingGroupName': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'AutoScalingGroupARN': 'arn:aws:autoscaling:ca-central-1:744734775600:autoScalingGroup:e18ade5c-2740-4648-85b5-d7d6b3253629:autoScalingGroupName/eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'MixedInstancesPolicy': {'LaunchTemplate': {'LaunchTemplateSpecification': {'LaunchTemplateId': 'lt-0fda9790b0b400c88', 'LaunchTemplateName': 'eks-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'Version': '1'}, 'Overrides': [{'InstanceType': 't3.xlarge'}]}, 'InstancesDistribution': {'OnDemandAllocationStrategy': 'prioritized', 'OnDemandBaseCapacity': 0, 'OnDemandPercentageAboveBaseCapacity': 0, 'SpotAllocationStrategy': 'capacity-optimized'}}, 'MinSize': 0, 'MaxSize': 10, 'DesiredCapacity': 0, 'DefaultCooldown': 300, 'AvailabilityZones': ['ca-central-1d', 'ca-central-1b', 'ca-central-1a'], 'LoadBalancerNames': [], 'TargetGroupARNs': [], 'HealthCheckType': 'EC2', 'HealthCheckGracePeriod': 15, 'Instances': [], 'CreatedTime': datetime.datetime(2022, 10, 11, 20, 29, 41, 507000, tzinfo=tzutc()), 'SuspendedProcesses': [], 'VPCZoneIdentifier': 'subnet-0286d381da9f5fa9c,subnet-0fc58322f447bae71,subnet-0381004140a514844', 'EnabledMetrics': [{'Metric': 'GroupStandbyCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupPendingInstances', 'Granularity': '1Minute'}, {'Metric': 'GroupPendingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTerminatingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTotalCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupTerminatingInstances', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolMinSize', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolTerminatingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupInServiceInstances', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolWarmedCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupMaxSize', 'Granularity': '1Minute'}, {'Metric': 'GroupTotalInstances', 'Granularity': '1Minute'}, {'Metric': 'GroupInServiceCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupAndWarmPoolDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupAndWarmPoolTotalCapacity', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolPendingCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupDesiredCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupMinSize', 'Granularity': '1Minute'}, {'Metric': 'WarmPoolTotalCapacity', 'Granularity': '1Minute'}, {'Metric': 'GroupStandbyInstances', 'Granularity': '1Minute'}], 'Tags': [{'ResourceId': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'ResourceType': 'auto-scaling-group', 'Key': 'eks:cluster-name', 'Value': 'dev-build', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'ResourceType': 'auto-scaling-group', 'Key': 'eks:nodegroup-name', 'Value': 'dip-jobs-2022101120290341260000000d', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'ResourceType': 'auto-scaling-group', 'Key': 'k8s.io/cluster-autoscaler/dev-build', 'Value': 'owned', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'ResourceType': 'auto-scaling-group', 'Key': 'k8s.io/cluster-autoscaler/enabled', 'Value': 'true', 'PropagateAtLaunch': True}, {'ResourceId': 'eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0', 'ResourceType': 'auto-scaling-group', 'Key': 'kubernetes.io/cluster/dev-build', 'Value': 'owned', 'PropagateAtLaunch': True}], 'TerminationPolicies': ['AllocationStrategy', 'OldestLaunchTemplate', 'OldestInstance'], 'NewInstancesProtectedFromScaleIn': False, 'ServiceLinkedRoleARN': 'arn:aws:iam::744734775600:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling', 'CapacityRebalance': True}], 'ResponseMetadata': {'RequestId': 'e4d9be35-53d6-4efe-b66e-bc6ad1f91a19', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'e4d9be35-53d6-4efe-b66e-bc6ad1f91a19', 'content-type': 'text/xml', 'content-length': '17188', 'vary': 'accept-encoding', 'date': 'Wed, 19 Oct 2022 20:17:05 GMT'}, 'RetryAttempts': 0}}

My end goal is to extract the AutoScalingGroupName and DesiredCapacity for each of the 2 list objects that are inside 1 list thats inside the main dict object.

The code I have till now is

    for key,value in response.items():
        for k in value:
            for keyy,valuee in k.items():
                if keyy == 'AutoScalingGroupName':
                    print(valuee)
                elif keyy == 'DesiredCapacity':
                    print(valuee)

This gives me

eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241
3
eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0
0
Traceback (most recent call last):
  File "/Users/me/Work/gitlab/img-asg-exporter/aws.py", line 62, in <module>
    get_asg_names()
  File "/Users/me/Work/gitlab/img-asg-exporter/aws.py", line 55, in get_asg_names
    for keyy,valuee in k.items():
AttributeError: 'str' object has no attribute 'items'

I understand why the error occurs and i know how to fix that. My question is whether this is the most optimal way to extract values like this ? I eventually want to append the values to a new dict and have it like

final = {
  "eks-core-20221011202902697400000006-6ac1e45e-44ee-4a43-166c-12d3d432e241": "3",
  "eks-dip-jobs-2022101120290341260000000d-9ec1e45e-4631-847b-b82d-d42d0fc330f0": "0"
}

>Solution :

There’s a lot of complication in your question to solve what seems like a relatively simple problem.

You appear to be asking: how do I get the 'AutoScalingGroupName' and 'DesiredCapacity' values from the dictionaries in the 'AutoScalingGroups' list in in my main dictionary response, and store them in a new dictionary structured as name: capacity?

So, that would simply be:

result = {
    asg['AutoScalingGroupName']: asg['DesiredCapacity'] 
    for asg in response['AutoScalingGroups']
}

The types and values of the other parts of the response dictionary are immaterial, as is the actual number of groups. As long as each auto-scaling group has a groupname and a capacity, the above will give you what you need.