subprocess.pipe function

[tebeka]

BPO	3548
Nosy	@amauryfa, @pitrou, @vapier, @merwok, @bitdancer, @desbma, @matheusportela
Files	pipe.patch: Patch file for "subprocess.pipe" toto.py: alternate implementation (not a patch) pipeline.py: clean, doc & unit test (still not a patch) pipeline.py: new version (pylint, license, copyright)

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-08-12.17:23:58.445>
labels = ['type-feature', 'library']
title = 'subprocess.pipe function'
updated_at = <Date 2020-03-05.02:01:55.533>
user = 'https://github.com/tebeka'

bugs.python.org fields:

activity = <Date 2020-03-05.02:01:55.533>
actor = 'vapier'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2008-08-12.17:23:58.445>
creator = 'tebeka'
dependencies = []
files = ['11104', '11262', '11263', '11330']
hgrepos = []
issue_num = 3548
keywords = ['patch']
message_count = 18.0
messages = ['71062', '71063', '71978', '71979', '72257', '72262', '72318', '114730', '122557', '125226', '249120', '249124', '249222', '249232', '249239', '249241', '249254', '363412']
nosy_count = 11.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'vapier', 'vlegoll', 'eric.araujo', 'vincent.legoll', 'r.david.murray', 'aht', 'Netto', 'desbma', 'matheus.v.portela']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3548'
versions = ['Python 3.3']

Linked PRs

• gh-47798: Add a subprocess.run_pipeline() API #142080

[tebeka]

Attached is a patch that add "pipe" command to the "subprocess" module.

pipe(["ls"], ["grep", "test_"]) will return the output of
"ls | grep test_".

[tebeka]

Not sure about the name, maybe "chain" will be better?

[vincentlegoll]

Hello,

I was searching for a bug in subprocess
module when I saw your patch.

I was implementing the exact same functionality
and mixed some of your ideas in what I use now,
which is attached...

Feel free to use it

[vincentlegoll]

Here's a clean version with doc & test

enjoy !

[vincentlegoll]

• Added "shut pylint up" comment for ** keyword expansion
• Added Copyright & license header

[amauryfa]

Vincent,
GPL licenced code is incompatible with the inclusion into python.
And if I am correct, you should sign a contributor agreement. Then the
licence text is not necessary.

[vlegoll]

On Mon, Sep 1, 2008 at 5:13 PM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Vincent,
GPL licenced code is incompatible with the inclusion into python.
And if I am correct, you should sign a contributor agreement. Then the
licence text is not necessary.

This is not a patch against python, this is a standalone script, just
implementing the same functionality as the original bug report in a
slightly different way...

But thanks for the input anyways.

If this functionality is really interesting people and agreed to be integrated
into upstream subprocess module, I can rework it the right way, or work
from the original patch from the bug report.

[aht]

I've written a package which can do this with arbitrary redirection in all subcommands (and some more).

You can, for example, do this:

    >>> Pipe(Sh('echo -n foo >&2', {2: 1}), Sh('cat; echo ERROR >&2', {2: os.devnull})).capture(1).read()
    'foo'

The package is at: http://github.com/aht/pc.py

[merwok]

pipe.patch looks interesting to me. I would replace **kwargs with a keyword-only argument named stderr, since that’s the only key used. This requires more tests and docs.

[pitrou]

I think this would be more useful if you could pass an optional input string (as in communicate()) and if it returned a (stdout, stderr) tuple. Or perhaps even a (return code, stdout, stderr) tuple; alternately, non-zero return codes could raise an exception.

[matheusportela]

I just found this open issue and I can work on it. What is left to do before closing it?

[bitdancer]

Thanks for being willing to work on it.

If what is wanted is a way to pipeline shell commands, Python already has that functionality in the pipes module.

So the interesting thing here would be pipelining *non* shell commands, to avoid the shell exploits that using a shell pipeline invites.

The pipes module already has a worked out API, so perhaps it would be useful to see about re-implementing pipe's command execution using subprocess, and expand the API to allow for argv style command specification that would be fed to subprocess using the default shell=False. This would also presumably allow pipes to be used when there's no bash shell available.

The downside is that we might break current uses of pipes if we replace the shell version of the pipelining with subprocess shell=True, while using subprocess only if the command specifications are argv lists would result in code with a split personality. But if I were working on it I'd experiment with that approach to see if it made sense.

Other core devs may have other opinions on this :)

[matheusportela]

Let me check whether I understood you suggestion...

What you are saying is that it is already possible to pipeline shell commands through subprocess, such as in the following line?

subprocess.call('ls -l | grep Music', shell=True)

However, this requires the command to be executed in a shell environment. Hence, it would be a good idea to extend it to non-shell command execution. Is this right?