Generator Tricks for Systems Programmers - Version 2.0

Presented at PyCon UK 2008, September 12, 2008.

(7/19/2009) The Python Essential Reference, 4th Edition is now available. If you like this tutorial, you'll like this edition--there is extended coverage of generators, coroutines, and other advanced Python features.

Introduction

• Presentation Slides (PDF)

Support Data Files

• generators-uk.zip

Code Samples

Part 2 : Processing Data Files

• nongenlog.py. Calculate the number of bytes transferred in an Apache server log using a simple for-loop. Does not use generators.

• genlog.py. Calculate the number of bytes transferred in an Apache server log using a series of generator expressions.

• makebig.py. Make a large access-log file for performance testing. This will create a file "big-access-log". For the numbers used in the presentation, I used python makebig.py 2000.

• genfind.py. A generator function that yields filenames matching a given filename pattern.

• genopen.py. A generator function that yields open file objects from a sequence of filenames.

• gencat.py. A generator function that concatenates a sequence of generators into a single sequence.

• gengrep.py. A generator that greps a series of lines for those that match a regex pattern.

• bytesgen.py. Example that finds out how many bytes were transferred for a specific file in a whole directory of log files.

• retuple.py. Parse a sequence of lines into a sequence of tuples using regular expressions.

• redict.py. Parse a sequence of lines into a sequence of dictionaries with named fields.

• fieldmap.py. Remap fields in a sequence of dictionaries.

• linesdir.py. Generate lines from files in a directory.

• apachelog.py. Parse an Apache log file.

• query404.py. Find the set of all documents that are broken (404).

• largefiles.py. Find all requests that transferred over a megabyte.

• largest.py. Find the largest document.

• hosts.py. Find unique host IP addresses.

• downloads.py. Find number of downloads of a specific file.

• robots.py. Find out who has been hitting robots.txt.

• robotsfast.py. Find out who has been hitting robots.txt (faster version).

• follow.py. Follow a log-file in real-time like tail -f in Unix. To run this program, you need to have a log-file to work with. Run the program runservers.py to start a simulated web-server. This will write a series of log lines for you to follow.

• realtime404.py. Print all 404 requests as they happen in real-time on a log file.

• genreceive.py. Generate clients that connect to a TCP socket.

• genmessages.py. Generate UDP messages.

• genpickle.py. Turn sequences of objects into a sequence of pickles.

• sendto.py. Send a sequence of items to a remote machine via a socket. Uses genpickle above.

• receivefrom.py. Receive a sequence of items from a socket. Uses genpickle above.

• genqueue.py. Consume items on a queue.

• genmultiplex.py. Multiplex many different generators into a single real-time stream using threads.

• broadcast.py. Broadcast a sequence of items to a collection of consumers.

• netsend.py. Send items to another host on the network. Requires a receiver (use receivefrom.py above).

• consthread.py. Broadcasting a generator to a collection of consumer threads.

• gentrace.py. Example of debugging a generator component.

• storelast.py. Store the last value of a generator (for access later in the processing pipeline)

• genshutdown.py. Simple example of shutting down a generator.

• shutdownevt.py. Shutting down a generator with an event.

• genrecord.py. Generate a sequence of records from a binary file.

• print_count.py. Example of using yield as a print statement.

• recvcount.py. A co-routine example.

• consumer.py. Co-routine example with a consumer decorator.

• logcoroutine.py. Processing log files with co-routines.