python subrpocess.Popen with pipe and stdout redirected to a file in the command not working

Here is the problem

Let’s consider a file :

printf ‘this is \n difficult’ >test

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

Now I would like to use python with the following bash command:

grep ‘diff’ test |gzip >test2.gz

I have tried the following code which didn’t work :

import subprocess

command = [‘grep’, ‘diff’, ‘test’, ‘|’, ‘gzip’, ‘>’ ‘test2.gz’]

proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, encoding=’utf8′, shell=True)

I have then tried to redirect the output to a file using:

import subprocess

command = [‘grep’, ‘diff’, ‘test’, ‘|’, ‘gzip’]

test2 = open(‘test2.gz’, ‘w’)

proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=test2, stderr=subprocess.STDOUT, encoding=’utf8′, shell=True)

But it didn’t work too so I am a bit clueless on how to proceed to be able to pipe and redirect to a file.

Best regards,

>Solution :

This doesn’t make any sense:

command = ['grep', 'diff', 'test', '|', 'gzip']

This is attempting to run the grep command with arguments ["diff", "test", "|", "gzip"] — which is not what you want — but when using shell=True you need to pass in a string rather than a list.

If you want to use shell scripting features like io redirection, you need to run a shell script.

import subprocess

command = "grep 'diff' test | gzip"

with open('test2.gz', 'w') as test2:
  proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=test2, stderr=subprocess.STDOUT, encoding='utf8', shell=True)