separate a dictionary of lists into chunks in Python

How can I split a dictionary of lists into chunks of a given size in Python. I can chunk a list. I can chunk a dictionary. But I can’t quite work out how to chunk a dictionary of lists (efficiently).

my_dict = {
  "key1": ["a","b","c","d"],
  "key2": ["e","f","g","h"],
}

How can I chunk it so that each chunk has no more than 3 values:

{
  "key1": ["a","b","c"]
}
{
  "key1": ["d"],
  "key2": ["e","f"],
}
{
  "key2": ["g","h"],
}

Notice how the 2nd dictionary spans 2 keys.

>Solution :

Quite a naive generator to do that would be:

1. Initialize a dict and a counter.
2. Iterate over all values (that is, all values inside the lists which are the dict’s values).
3. Add them to the new dict with the matching key, and increment the counter.
4. Once the required size is reached, yield/return the chunk and re-initialize the dict and counter.
5. back to 2.

That would be:

from collections import defaultdict

def chunker(d, chunk_size):
    chunk = defaultdict(list)
    size = 0
    for key, list_value in d.items():
        for value in list_value:
            chunk[key].append(value)
            size += 1
            if size == chunk_size:
                yield dict(chunk)
                chunk = defaultdict(list)
                size = 0
    if size:
        yield dict(chunk)

And running:

my_dict = {
  "key1": ["a","b","c","d"],
  "key2": ["e","f","g","h"],
}

for chunk in chunker(my_dict, 3):
    print(chunk)

Will give:

{'key1': ['a', 'b', 'c']}
{'key1': ['d'], 'key2': ['e', 'f']}
{'key2': ['g', 'h']}

Instead of a simple dict, this uses a defaultdict to simplify the code.