Metaclasses in Python 1.5

Linux 4gvps.4gvps.com 3.10.0-1127.18.2.vz7.163.46 #1 SMP Fri Nov 20 21:47:55 MSK 2020 x86_64
SOFT : Apache PHP : 7.4.33
/usr/lib64/python2.7/Demo/metaclasses/

38.135.39.45

[ NAME ]	[ SIZE ]	[ PERM ]	[ DATE ]	[ ACT ]
+FILE +DIR
Eiffel.py	3.387 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Eiffel.pyc	4.523 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Eiffel.pyo	4.424 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Enum.py	4.315 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Enum.pyc	6.034 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Enum.pyo	6.034 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Meta.py	2.864 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Meta.pyc	4.472 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Meta.pyo	4.472 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Simple.py	1.164 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Simple.pyc	2.682 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Simple.pyo	2.682 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Synch.py	7.752 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Synch.pyc	8.264 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Synch.pyo	8.119 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Trace.py	4.036 KB	-rw-r--r--	2013-05-12 03:32	R E G D
Trace.pyc	6.028 KB	-rw-r--r--	2024-10-03 12:56	R E G D
Trace.pyo	6.028 KB	-rw-r--r--	2024-10-03 12:56	R E G D
index.html	20.093 KB	-rw-r--r--	2013-05-12 03:32	R E G D
meta-vladimir.txt	11.257 KB	-rw-r--r--	2013-05-12 03:32	R E G D
REQUEST EXIT

(A.k.a. The Killer Joke :-)

In previous Python releases (and still in 1.5), there is something called the ``Don Beaudry hook'', after its inventor and champion. This allows C extensions to provide alternate class behavior, thereby allowing the Python class syntax to be used to define other class-like entities. Don Beaudry has used this in his infamous MESS package; Jim Fulton has used it in his Extension Classes package. (It has also been referred to as the ``Don Beaudry hack,'' but that's a misnomer. There's nothing hackish about it -- in fact, it is rather elegant and deep, even though there's something dark to it.)

(On first reading, you may want to skip directly to the examples in the section "Writing Metaclasses in Python" below, unless you want your head to explode.)

Documentation of the Don Beaudry hook has purposefully been kept minimal, since it is a feature of incredible power, and is easily abused. Basically, it checks whether the type of the base class is callable, and if so, it is called to create the new class.

(Types are not classes, by the way. See questions 4.2, 4.19 and in particular 6.22 in the Python FAQ for more on this topic.)

So our conclusion is that in our example, the type of the base class (of C) is not callable. So the Don Beaudry hook does not apply, and the default class creation mechanism is used (which is also used when there is no base class). In fact, the Don Beaudry hook never applies when using only core Python, since the type of a core object is never callable.

So what do Don and Jim do in order to use Don's hook? Write an extension that defines at least two new Python object types. The first would be the type for ``class-like'' objects usable as a base class, to trigger Don's hook. This type must be made callable. That's why we need a second type. Whether an object is callable depends on its type. So whether a type object is callable depends on its type, which is a meta-type. (In core Python there is only one meta-type, the type ``type'' (types.TypeType), which is the type of all type objects, even itself.) A new meta-type must be defined that makes the type of the class-like objects callable. (Normally, a third type would also be needed, the new ``instance'' type, but this is not an absolute requirement -- the new class type could return an object of some existing type when invoked to create an instance.)

Still confused? Here's a simple device due to Don himself to explain metaclasses. Take a simple class definition; assume B is a special class that triggers Don's hook:

In either case, the creator is called with three arguments. The first one, name, is the name of the new class (as given at the top of the class statement). The bases argument is a tuple of base classes (a singleton tuple if there's only one base class, like the example). Finally, namespace is a dictionary containing the local variables collected during execution of the class statement.

Note that the contents of the namespace dictionary is simply whatever names were defined in the class statement. A little-known fact is that when Python executes a class statement, it enters a new local namespace, and all assignments and function definitions take place in this namespace. Thus, after executing the following class statement:

But enough already about writing Python metaclasses in C; read the documentation of MESS or Extension Classes for more information.

Writing Metaclasses in Python

In Python 1.5, the requirement to write a C extension in order to write metaclasses has been dropped (though you can still do it, of course). In addition to the check ``is the type of the base class callable,'' there's a check ``does the base class have a __class__ attribute.'' If so, it is assumed that the __class__ attribute refers to a class.

At this point it may be worth mentioning that C.__class__ is the same object as B.__class__, i.e., C's metaclass is the same as B's metaclass. In other words, subclassing an existing class creates a new (meta)inststance of the base class's metaclass.

Going back to the example, the class B.__class__ is instantiated, passing its constructor the same three arguments that are passed to the default class constructor or to an extension's metaclass: name, bases, and namespace.

It is easy to be confused by what exactly happens when using a metaclass, because we lose the absolute distinction between classes and instances: a class is an instance of a metaclass (a ``metainstance''), but technically (i.e. in the eyes of the python runtime system), the metaclass is just a class, and the metainstance is just an instance. At the end of the class statement, the metaclass whose metainstance is used as a base class is instantiated, yielding a second metainstance (of the same metaclass). This metainstance is then used as a (normal, non-meta) class; instantiation of the class means calling the metainstance, and this will return a real instance. And what class is that an instance of? Conceptually, it is of course an instance of our metainstance; but in most cases the Python runtime system will see it as an instance of a a helper class used by the metaclass to implement its (non-meta) instances...

Hopefully an example will make things clearer. Let's presume we have a metaclass MetaClass1. It's helper class (for non-meta instances) is callled HelperClass1. We now (manually) instantiate MetaClass1 once to get an empty special base class:

We are now ready to create instances of MySpecialClass. Let's assume that no constructor arguments are required:

Now let's see how we could use metaclasses -- what can we do with metaclasses that we can't easily do without them? Here's one idea: a metaclass could automatically insert trace calls for all method calls. Let's first develop a simplified example, without support for inheritance or other ``advanced'' Python features (we'll add those later).

The class Instance is the class used for all instances of classes built using the Tracing metaclass, e.g. aninstance. It has two methods:

The __getattr__ method looks the name up in the __namespace__ dictionary. If it isn't found, it raises an AttributeError exception. (In a more realistic example, it would first have to look through the base classes as well.) If it is found, there are two possibilities: it's either a function or it isn't. If it's not a function, it is assumed to be a class variable, and its value is returned. If it's a function, we have to ``wrap'' it in instance of yet another helper class, BoundMethod.

The BoundMethod class is needed to implement a familiar feature: when a method is defined, it has an initial argument, self, which is automatically bound to the relevant instance when it is called. For example, aninstance.method1(10) is equivalent to method1(aninstance, 10). In the example if this call, first a temporary BoundMethod instance is created with the following constructor call: temp = BoundMethod(method1, aninstance); then this instance is called as temp(10). After the call, the temporary instance is discarded.

In order to be able to support arbitrary argument lists, the __call__ method first constructs a new argument tuple. Conveniently, because of the notation *args in __call__'s own argument list, the arguments to __call__ (except for self) are placed in the tuple args. To construct the desired argument list, we concatenate a singleton tuple containing the instance with the args tuple: (self.instance,) + args. (Note the trailing comma used to construct the singleton tuple.) In our example, the resulting argument tuple is (aninstance, 10).

The intrinsic function apply() takes a function and an argument tuple and calls the function for it. In our example, we are calling apply(method1, (aninstance, 10)) which is equivalent to calling method(aninstance, 10).

From here on, things should come together quite easily. The output of the example code is something like this:

That was about the shortest meaningful example that I could come up with. A real tracing metaclass (for example, Trace.py discussed below) needs to be more complicated in two dimensions.

First, it needs to support more advanced Python features such as class variables, inheritance, __init__ methods, and keyword arguments.

Second, it needs to provide a more flexible way to handle the actual tracing information; perhaps it should be possible to write your own tracing function that gets called, perhaps it should be possible to enable and disable tracing on a per-class or per-instance basis, and perhaps a filter so that only interesting calls are traced; it should also be able to trace the return value of the call (or the exception it raised if an error occurs). Even the Trace.py example doesn't support all these features yet.

Real-life Examples

Have a look at some very preliminary examples that I coded up to teach myself how to write metaclasses:

A pattern seems to be emerging: almost all these uses of metaclasses (except for Enum, which is probably more cute than useful) mostly work by placing wrappers around method calls. An obvious problem with that is that it's not easy to combine the features of different metaclasses, while this would actually be quite useful: for example, I wouldn't mind getting a trace from the test run of the Synch module, and it would be interesting to add preconditions to it as well. This needs more research. Perhaps a metaclass could be provided that allows stackable wrappers...

Things You Could Do With Metaclasses

There are lots of things you could do with metaclasses. Most of these can also be done with creative use of __getattr__, but metaclasses make it easier to modify the attribute lookup behavior of classes. Here's a partial list.

Credits

Many thanks to David Ascher and Donald Beaudry for their comments on earlier draft of this paper. Also thanks to Matt Conway and Tommy Burnette for putting a seed for the idea of metaclasses in my mind, nearly three years ago, even though at the time my response was ``you can do that with __getattr__ hooks...'' :-)