I’ve observed that the RAM holds on to data even after stopping/quitting the spark-shell instance. How can I free RAM memory?

>Solution :

Even after you quit your spark-shell instance, the spark application will still be running. For the RAM to be released, you can try killing the application instance either using the web UI end point or directly on your shell

You can find the application process by the following

ps -aef|grep spark-shell

Once you find the process, then you can kill it by

kill -9 pid