Version 0.8 of stream.py is out with a lot of goodies to make it easy to parallelize a pipeline. I have written previously that by expressing a list-processing task as a pipeline (or at least thinking of it as such), we can have parallelism for free. Well, at least as free as Python allows us to be. We can choose between threading and multiprocessing as our concurrency model: with first choice, we have shared-memory but no real concurrence of threads (thanks to the GIL), and with second choice, we have real multi-tasking but only under the tax of ...
Parallelized list-processing with stream.py
16.01.2010
| Filed under
python,
article,
concurrency,
stream.py
Streamlined list processing aka. pipes in Python
15.12.2009
| Filed under
python,
article,
stream.py
One of the most widely admired contributions of Unix to the culture of operating systems and command languages is the pipe, as used in a pipeline of commands. — Dennis M. Ritchie [dmr]
If the pipe is so great; why, then, does your favorite modern, dynamic programming language not have pipes built-in?
If the idea is merely to direct the output of some computation to be the input of another, any programming languages with a procedure abstraction would let you do that simply by composition. Like this,
f(g(h(input)))
However, that is not a pipeline! The procedure f can ...