I’ve observed that the RAM holds on to data even after stopping/quitting the spark-shell instance. How can I free RAM memory?
>Solution :
Even after you quit your spark-shell instance, the spark application will still be running. For the RAM to be released, you can try killing the application instance either using the web UI end point or directly on your shell
You can find the application process by the following
ps -aef|grep spark-shell
Once you find the process, then you can kill it by
kill -9 pid