In this talk I present a war-story about a 300-line if-statement. I talk about how the code came about, and how I wrangled it to something that could be managed more easily.

For those new to Python I present some techniques, idioms, and libraries for managing data. For more experienced Pythonistas I will demonstrate a way to implement some patterns, and other useful technique to decompose software into smaller parts.

The if-statement in question lurked deep within the code-base of GroupServer ― a mailing-list manager with a great web interface.

• With a mailing-list manager you normally define a group, like GroupServer development.
• A normal group works like this:
  1. A member sends an email to the list,
  2. It is processed by GroupServer, and
  3. GroupServer sends an email to all the group members.

• Only members can post to (most) lists.
• People who are not members are told to go away.

It is this if-statement that got away on us.

The pseudo-code for the if-statement on the previous two slides is not too terrible. It has two branches:

1. Send the email out if the message is from a member.
2. Notify everyone else that they cannot post.

However, when you have code being used by people complexity has a habit of rising, a bit like damp. Or termites.

The pseudo-code for a more realistic scenario would be a bit more like this, with four different types of groups.

• Our if-statement from before is still there, at the top. It applies to discussion groups only.
• GroupServer introduced announcement groups, like GroupServer announcements, that restrict posting to the subset of Posting members.
  • Posting members can post,
  • Most group members are blocked from posting, and
  • Non-members are also blocked from posting.
• Support groups
  • Support groups are needed because of the complexity of the other two groups!
  • Everyone can post to a support-group.
  • Only group members receive the messages.
• Closed groups
  • No one can post to a closed group.
  • GroupServer has them to allow the archive to access the excellent archives, even after the group has finished.
  • Also used before the group starts, so there can be a good launch.

Experts may start feeling twitchy about now, but it does not end there…

Have not even covered all the other rules that have nothing to do with membership.

• Some people horrid, so are blocked.
• There is a difference in how GroupServer responds to people with and without a profile.
• Broken email addresses need to be addressed specifically.
• Those with a profile can lack required fields (that may be specific to a group).
• The posting-limit is there to make people write thoughtful email messages.

And all this ends up leading to…

The 300-line if-statement.

• Not supposed to be able to read this: it is embarrassing enough already.
• This gives you an idea of what the file felt like.
• It lacks the closed group-type. My boss asked me to add closed groups, and I declined, because I could not touch the code without breaking it. So he allowed me to rewrite the code into something more sane.

This is the current code at the same typeface size as the previous slide, to give you some idea of the change.

• On the left there are about a dozen lines, with a Copyright and licence block, and a few imports
• On the right there are another two-dozen lines of actual code. Where did the rules go?
  • Each rule is in a class,
  • Rules assembled by this class,
  • A single aggregate Boolean is delivered by this code.

The first detailed bit of code I want to look at is the one that returns the Boolean, canPost

• The core logic is a reduce that does an and across all the rules, so it works like an if.
• Lot of useful things from core Python here. (This is both Python 2, Python 3, PyPy, and PyPy3 code.)
  • @property is a decorator. It makes a method behave like a property, so you can call it like obj.canPost ― without the brackets.
  • The functools standard library contains many things for dealing with lists-of-stuff, a style of programming known as functional programming, hence the name. In this case I am using the reduce function to reduce a list to a single value. It takes three arguments, but the two main ones are what to reduce, and how to reduce it.
    1. What to reduce, in this case a list of rules. We create a list of Boolean values by asking each rule if the person can post, using a list comprehension.
    2. How to reduce this list. In this case I am doing a logical and using operator.and_. The operator library is the off-sider of functools. It contains all the Python operators as functions, removing the need for a lambda function.
    3. Finally there is an initial value, which is used if the list is empty or has only one value.

The second part of the can-post code gets the rules that are used in the canPost method.

This is the code that gets all the rules, which are fed to the reduce we saw previously. It contains some standard things: the @property decorator, the use of a list comprehension, the sort method of a list, and operator.attrgetter (the operator library once-again saving us a lambda).

Then there are some non-core bits, which I will explain soon, but I will point out the bits you have probably never seen:

• There is this registry thing. It is a singleton, like a database we do not want more than one copy of a registry of things.
• And then there is this getAdapters call. It is passed a 2-tuple of a user and a group. The user-and-group had to come up some time as we are trying to figure out if a user can post to a group. Then there is what looks like a class-name, beginning with I.

To explain this code I have to explain what an adapter is.

Conceptually an adapter is what it sounds like:

• Take something with one interface (110V, 10A, AC)
• Munge it some how
• Provide something with a different interface (5V, 0.85A, DC)

However, I will not discuss such simple adapters today.

What we are dealing with is a more complex multi-adapter.

• Take two or more inputs: a DSL line connecting to the outside World, and 12V DC power.
• Provide something with another interface: Ethernet.

In our case we are adapting a user and a group, in order to get a rule. So what we are adapting is passed in as the first argument, which is why it is in a tuple. What we want is the last (second) argument.

So what does one of these adapters look like?

Adapters look like normal classes. Here is the actual rule for an announcement group (almost).

• There is the weight that the rules were sorted by.
• It is initialised with a user and a group, which was passed in by the getAdapters call
• The canPost attribute is following my normal style of using a method with the @property decorator. (The rule is looking up a list of user-identifiers.)

But what makes this an adapter rather than a normal class?

All these extra things that I hid from you make this an adapter. They may look odd, but they are just signals about what this class does. If we look at this new code a bit closer…

• The @adapter is a class decorator. It mirrors the user and group ― both in the __init__ and in the getAdapters call. However, we have more of these funny class names beginning with I that I promise to get to soon.
• Another class-decorator @implementer now closely matches up to something in the getAdapters call.

However, declaring these is not enough, we also have to keep track of them. Somehow.

Keeping track of the adapters is where the registry, which you saw earlier, comes into play again. It keeps track of everything for us. The only oddity is calling the getGlobalSiteManager each time to get it: it returns a singleton, because we only want one single registry.

The adapters themselves are added using registerAdapter, which takes a class and a name (which you may recall we threw out earlier).

To explain why it is called a site manager, and where these decorators and functions come from, I must address the elephant in the room.

• The three non-standard things that I have shown to you (and there are only three) have all come from two libraries: zope.component and zope.interface.
• Together, with another small library called zope.event, they are known as the Zope Component Architecture (ZCA).
• Zope has a… patchy representation, which unfairly sullies the reputation of the ZCA: a friend of mine once quipped that the name Zope was the worst thing about the ZCA. Despite this, the ZCA provides the core of Pyramid, Plone, and Mailman 3.
• The web-framework is still present, but it is totally optional. I am just going to talk about the ZCA.

And finally, I guess I should talk about these I things. You see one here: Interface.

As a concept interfaces are similar to interfaces in strongly typed languages such as Java, and they can be used to specify attributes and methods. However in practice I just use them as a sign (MasterCard accepted here).

• The interfaces themselves are just classes that inherit from zope.interface.Interface and contain a doc-string. I use duck-typing, as normal in Python, for actual conformance to an API.
• In UML they have the interface stereotype, and tradition has it that an interface class starts with I ― so they do not get mixed up with normal classes.
• Classes that use interfaces are drawn with lolly-pop connectors in UML:
  • Take something with a user-interface and announcement group interface.
  • Provide something with a can-post rule interface.
  • The classes that use interfaces remain unconnected: they just look like they can connect. That is because this UML only shows the static structure, and the rule-instances were only connected up by getAdapters at run time.

However, interface classes themselves do get some UML connection action on.

• Interfaces can inherit, so they can be more specific: HSBC MasterCard accepted here.
• All eventually inherit from the base Interface.
• With our running announcement group example, its interface inherits from the discussion group interface.
• The other group-interfaces just inherit from the base group interface.

Because interfaces inherit a really cool thing happens when we ask for all the rules for an announcement group.

• The can-post code gets a bunch of adapters for a user and a group. The can post code is unconcerned about the specifics of the what the user and group are, just that it gets rules rules.
  • There is one rule for an announcement group, that for the posting members, so the can-post code will get that.
  • It also gains six rules because the announcement group interface inherits from the discussion group.
  • The discussion group inherits from the base group, so we get a rule from that, too.
• Finally all the rules are put together with an and_.

Now, you could argue that we have a complex system, and you'd be right.

The complexity of reality is intractable: discussion groups, announcement groups, support groups, and closed groups are all needed. The system has to reflect reality. However, using the ZCA we can create small components that are easy to understand.

• Less likely to make an error because it is understandable in isolation.
• The components are highly cohesive, in they work well together.
• The components are lightly coupled, in you can change one without changing the others. If the Python product that supplies the Announcement group (gs.group.type.announcement) is not loaded then everything else is fine.
• Being small and lightly coupled makes things easier to test, as each rule can be tested independently. Testing the old code would have sent the best programmer mad just with getting the mocking done.

And that'd be it, except for some messy confessions…

First, if you look at the actual GroupServer code you will notice an absence of @adapter and @implementer class decorators. That is because I use the Zope Configuration Markup Language (ZCML).

• An XML-based language that is used to provide the decoration and registration.
• I think it simplifies the code.
• I like how it places all the component-information in one file (configure.zcml)
• Some disagree about the value of the ZCML, and I am willing to accept that I may be one of the few people alive who likes it.

I actually use the @Lazy method decorator, rather than @property. In practice, it does much the same job, but it caches the result, so calling it the second time gives you an immediate answer. Switching methods and properties to @Lazy is a really cheep and easy way to speed up code.

There are other cache descriptors in the zope.cachedescriptors class, including ones where you can reset the cache. However, I have never needed anything other than @Lazy.

I have shown you many design patterns today, but the adapter pattern is not one of them.

Strategy

The Strategy pattern defines an interchangeable set of algorithms. This is the main pattern that we have been walking through.

Singleton

I have pointed out the singleton registry when it has popped up.

Abstract factory

The abstract factory pattern allows us to separate a concrete implementation from the code that uses it, thanks to interfaces. We used getAdapters to get a bunch of adapters, but you can get just one:

storage = getMultiAdapter(
    (groupInfo, msg), IStorageForEmailMessage)
storage.store()

Decorator

In design patterns a decorator enhances a class, but does not change the interface. In Python support for decorators is built-in, which we saw with @adapter and @implementer

@implementer(IGSCanPostRule)

Proxy

Defines a surrogate for another object, which we just saw with the @Lazy decorator that hides a complex operator behind a light-weight property.

I have been using the term adapter in the ZCA sense. So what about the actual adapter pattern?

With the adapter pattern you get a old-thing with a terrible API, and wrap it in a new class (the adapter) that has a nice API.

Raymond Hettinger gave a great talk at PyCon in 2015 where he created an adapter from scratch. The video is on YouTube, and it is worth watching simply for the performance-art aspect of the talk.

• GroupServer can be found on the web at GroupServer.org.
• You can also get the source-code from GitHub.
• Those that like to follow the thoughts of a mailing-list manager with an excellent web interface can follow GroupServer on Twitter.
• And finally, I am me. I like email.