Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Why does a dictionary of numpy arrays use less memory than an ndarray?

I am trying to find a memory efficient way to store data in python variables for quick access and analysis. I initialize an 2d array in numpy and then find its memory usage (using sys so I can compare to other variable types later) via the following:

a = np.zeros((1000,1000), dtype=np.float32)
print('The size of the numpy array is {} bytes'.format(sys.getsizeof(a)))

Which returns: The size of the numpy array is 4000112 bytes

I can move this into a dictionary of 1d numpy arrays using the following for-loop:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

b = {}
for ii in range(1000):
    b[f'{ii}']=a[:,ii]

print('The size of the dictionary is {} bytes'.format(sys.getsizeof(b)))

Which returns: The size of the dictionary is 36968 bytes. The dictionary size persists even if I delete a, so b can’t just be a container pointing to a.

Why would a dictionary of 1d arrays take up less memory than those same arrays in an ndarray?

>Solution :

There are two fundamental mistakes in your observation.

  1. you cannot delete an object, only references. If you delete a you delete the pointer. When you delete all pointers then only the object might get deleted at some point by the garbage collector

  2. sys.getsizeof only gives the size of the container. To get the total size you need to loop over the elements and sum.

Demonstration that the size is roughly the same:

b = {}
for ii in range(1000):
    b[f'{ii}']=a[:,ii].copy()
sum(sys.getsizeof(e) for e in b.values())
# 4096000
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading