Understanding Python's Metaclass-Based Metamodel: A Deep Dive

Introduction

While Python has established itself as the predominant language in AI and machine learning domains, developers coming from strongly-typed backgrounds often find aspects of Python's design worth critique. However, one absolute highlight—and personally the most appreciated feature—is Python's metaclass-based metamodel. This article explores the intricacies and elegance of Python's metamodel architecture.

1. The Metaverse Analogy

To understand metaclasses, consider the concept of "metaverse" that gained prominence recently. From a "creation" perspective, the metaverse defines the laws governing our universe (reality). The universe itself is an instance constructed from metadata.

The relationship between "meta" and "instance" isn't absolute—the metaverse is itself a universe, with its laws defined by a higher-order "meta-metaverse," making the metaverse an instance of this meta-metaverse. This abstraction continues until reaching an ultimate origin where no further law source exists. Some call this the "Creator," Laozi termed it "Dao." We might call this the "Genesis Universe."

Unable to find a meta for the Genesis Universe, we consider it its own meta, forming a self-consistent closed loop. Since we view the universe as an instance of its metaverse, the Genesis Universe's instances include itself.

Key Insight: Meta can be simpler than instances—"the Great Dao is simple." As the saying goes, "Dao generates One, One generates Two, Two generates Three, Three generates all things." Conversely, meta can also be more complex than instances, containing intricate laws from which we select subsets to construct instances.

In Python's "metaverse," we use meta to define rules for instances, serving as a factory for instance creation. Thus, classes are the meta of instances. We can use metaclasses to create classes, making regular classes instances of metaclasses. Metaclasses are the meta of regular classes. The term "metaclass" expresses both "class's meta" and the concept that metaclasses themselves are classes. Since metaclasses are classes, they can have their own metaclasses.

Fundamental Truth: Class is meta, meta is also class—just as "metaverse is universe, universe can serve as metaverse."

Python's Genesis Universe is type. Our classes are constructed by it, making type the metaclass of classes. Since type is a metaclass, it possesses class attributes. Occupying this transcendent position, type can be considered its own instance.

2. Metaclass Definition

By default, when creating an instance using a class, the underlying process follows:

1. Pass the class and parameters (including any keyword arguments) to the class's __new__ method to create a base object
2. Pass this base object and parameters to the class's __init__ method for initialization
3. Return the initialized object

Consider the following Foobar class definition. Two equivalent ways to create instances (foobar1 and foobar2) exist:

from typing_extensions import Self
from typing import Any

class Foobar:
    def __new__(cls, *args: Any, **kwargs: Any) -> Self:
        return super().__new__(cls)
    
    def __init__(self, foo: int, bar: int) -> None:
        self.foo = foo
        self.bar = bar
    
    def __eq__(self, value: object) -> bool:
        if not isinstance(value, Foobar):
            return NotImplemented
        return self.foo == value.foo and self.bar == value.bar

# Method 1: Standard instantiation
foobar1 = Foobar(foo=111, bar=222)

# Method 2: Explicit __new__ and __init__ calls
foobar2 = Foobar.__new__(Foobar, foo=111, bar=222)
Foobar.__init__(foobar2, foo=111, bar=222)

assert foobar1 == foobar2

Important Note: While __new__ appears like a class method in definition, it's本质上 a static method. It lacks the standard @classmethod decorator, and the first parameter must be explicitly specified during invocation.

Similarly, metaclasses, being classes themselves, can define __new__ and __init__ methods. These methods in regular classes initialize their instances; in metaclasses, their mission is identical—except the instances of a metaclass are the classes that use it as their meta.

When the Python interpreter encounters a class definition with a specified metaclass, it creates the class similarly:

1. Call the metaclass's __new__ method with fixed parameters to construct a class object:

   • Current metaclass
   • Class name
   • Class members (as a dictionary)
   • Keyword arguments specified during class definition

2. Call the metaclass's __init__ method to initialize the base class object:

   • self is the class object
   • Remaining parameters include class name, members dictionary, and keyword arguments

Practical Example

from typing import Any

class Meta(type):
    def __new__(cls, name: str, bases: tuple[type, ...], namespaces: dict[str, Any], /, **kwds: Any):
        print(f"""
__new__
 cls: {cls}
 name: {name}
 bases: {bases}
 namespaces: {namespaces}
 kwds: {kwds}
""")
        for key, value in kwds.items():
            namespaces[key] = value
        return super().__new__(cls, name, bases, namespaces)
    
    def __init__(self, name: str, bases: tuple[type, ...], dict: dict[str, Any], /, **kwds: Any) -> None:
        print(f"""
__init__
 cls: {self}
 name: {name}
 bases: {bases}
 dict: {dict}
 kwds: {kwds}
""")
        def repr(self) -> str:
            express = ", ".join(f"{key}={value!r}" for key, value in kwds.items())
            return f"({express.strip()})"
        setattr(self, "__repr__", repr)

class Foo:
    ...

class Bar(Foo, metaclass=Meta, x=-1, y=-1):
    ...

print(Bar())

Execution Output:

__new__
 cls: <class '__main__.Meta'>
 name: Bar
 bases: (<class '__main__.Foo'>,)
 namespaces: {'__module__': '__main__', '__qualname__': 'Bar', ...}
 kwds: {'x': -1, 'y': -1}

__init__
 cls: <class '__main__.Bar'>
 name: Bar
 bases: (<class '__main__.Foo'>,)
 dict: {'__module__': '__main__', '__qualname__': 'Bar', ..., 'x': -1, 'y': -1}
 kwds: {'x': -1, 'y': -1}

(x=-1, y=-1)

In this example, Meta serves as the metaclass for Bar, while Foo is Bar's base class. The two keyword arguments specified during Bar's definition become type field members per Meta's __new__ implementation.

3. Instances Are Created by Metaclasses

The previous section described instance creation flow: first calling __new__ to construct a base object, then passing it to __init__ for initialization. However, this is merely表象 (appearance), not essence. The fundamental truth is: instances of a class are created by its metaclass.

Viewing metaclasses as instance-creation factories, and recognizing that everything in Python is an object (including functions), we understand that functions are instances of the function class. The function class possesses a __call__ method (either self-defined or inherited). Calling a function is essentially invoking the function object's __call__ method.

When invoking a metaclass in function form to create instances, it means instances are created through the __call__ method defined in the metaclass:

from typing import Any

class Foo:
    ...

class Meta(type):
    def __call__(self, *args: Any, **kwds: Any) -> Any:
        assert self is Bar
        assert type(self) is Meta
        assert args == ("111", "222")
        assert kwds == {"c": "333", "d": "444"}
        return Foo()

class Bar(metaclass=Meta):
    def __init__(self, a: str, b: str, **kwargs) -> None:
        self.x = a
        self.y = b
        self.kwargs = kwargs

assert isinstance(Bar("111", "222", c="333", d="444"), Foo)

Analysis: As a method, the first parameter is always the calling subject object (class for class methods, instance for instance methods). The assertions in __call__ reveal that the calling subject is the Bar class object itself, and type(self) returns Meta—the metaclass defining this class. Positional and keyword arguments passed to Bar("111", "222", c="333", d="444") are assigned to args and kwds as tuple and dictionary respectively.

Question: Why does calling Foo() return a Foo object for regular classes like Foo?

Answer: The same rule applies. Although Foo doesn't specify a concrete metaclass, type serves as the fallback metaclass. The instance returned by Foo() is actually created by the __call__ method defined in type:

class type:
    def __call__(self, *args: Any, **kwds: Any) -> Any:
        instance = self.__new__(self, *args, **kwds)
        self.__init__(instance, *args, **kwds)
        return instance

This method creates objects by:

1. Calling the class's __new__ to construct a base object
2. Passing that object to the class's __init__ for initialization, then returning it

We needn't worry about whether a class defines __new__ or __init__—the ultimate base class object provides defaults:

• A parameterless __new__ allocates memory for an empty object
• A parameterless __init__ does nothing

4. The Ultimate Metaclass: type

4.1 type.__new__ Method

Let's examine the logic in type's __new__ method, used to construct class objects. Our custom metaclass Meta's __new__ ultimately calls this method (return super().__new__(cls)).

Method Signature:

class type:
    def __new__(
        cls: type[Self],
        name: str,
        bases: tuple[type, ...],
        namespace: dict[str, Any], /,
        **kwds: Any
    ) -> Self: ...

Five Parameters:

• cls: The metaclass constructing the class
• name: Names the ultimately constructed class
• bases: Base classes for the constructed type
• namespaces: Type members
• kwds: Additional keyword arguments

Scenario 1: If cls parameter is the type class object itself, a regular type is constructed—named by name, inheriting from bases, with members from namespaces. The kwds parameters are meaningless:

class Foo:
    x = -1

cls = type.__new__(type, "Bar", (Foo,), {"y": -1})
assert cls.__name__ == "Bar"
assert cls.__bases__ == (Foo,)
assert type(cls) is type

bar = cls()
assert bar.x == -1
assert bar.y == -1

Scenario 2: Specifying a custom metaclass for cls becomes interesting. Since both metaclasses and type's __new__ can create types, conflicts arise. The latter has higher priority (explicit call), so the custom metaclass's __new__ and __init__ won't execute:

class Baz:
    ...

log = []

class Meta(type):
    def __new__(cls, name: str, bases, namespaces, /, **kwds):
        log.append(f"Meta.__new__ is called")
        return Baz
    
    def __init__(self, *args, **kwargs):
        log.append(f"Meta.__init__ is called")
        self.z = -1

class Foo:
    x = -1

cls = type.__new__(Meta, "Bar", (Foo,), {"y": -1})

assert len(log) == 0  # Custom metaclass methods NOT called
assert cls.__name__ == "Bar"
assert cls.__bases__ == (Foo,)
assert type(cls) is Meta  # But metaclass IS set correctly

bar = cls()
assert bar.x == -1
assert bar.y == -1
assert not hasattr(bar, "z")  # __init__ didn't run

Although the metaclass's __new__ and __init__ don't execute, the specified metaclass is correctly set as the generated type's metaclass. If the metaclass overrides __call__, that method will be invoked during instantiation:

class Baz:
    ...

class Meta(type):
    def __call__(self, *args, **kwargs):
        return Baz()

class Foo:
    x = -1

cls = type.__new__(Meta, "Bar", (Foo,), {"y": -1})
assert isinstance(cls(), Baz)  # Custom __call__ IS used

4.2 The type Function

As the ultimate metaclass, type is also a class. When calling a class as an executable function, the metaclass's __call__ method is invoked. Since type's metaclass is itself, calling type() essentially invokes type's own __call__ method.

Behavior:

1. Single Object Argument: Returns the object's class

   • For regular objects: returns the object's class
   • For classes: returns the metaclass
   • If no explicit metaclass: returns the fallback metaclass type

class Meta(type):
    pass

class Foobar(metaclass=Meta):
    pass

assert type(Foobar()) is Foobar      # Instance's class
assert type(Foobar) is Meta          # Class's metaclass
assert type(type) is type            # type's own metaclass

2. Multiple Parameters: Creates a new class, requiring: class name, base classes, class members, keywords (matching __new__ parameters)

class Base():
    foo = -1

cls = type("Foobar", (Base,), {"bar": -1})
assert cls.__name__ == "Foobar"
assert cls.__bases__ == (Base,)
assert cls.bar == -1

instance = cls()
assert instance.foo == -1
assert instance.bar == -1

Thus, type's __call__ determines whether to return an object's class or create a new class based on argument format:

class type:
    def __call__(self, *args, **kwargs):
        # Single parameter: return object's class
        if len(args) == 1 and not kwargs:
            obj = args[0]
            return getattr(obj, "__class__", type(obj))
        
        # Multiple parameters: create new class
        instance = self.__new__(self, *args, **kwargs)
        self.__init__(instance, *args, **kwargs)
        return instance

5. Reorganizing Class Generation and Instantiation

Let's summarize class generation and instantiation based on classes:

For class code written with class keyword, Python interpreter:

1. Extracts the metaclass (defaults to type if not explicitly specified), calls the metaclass's __new__ with class definition info (name, bases, members, keywords):

   • For custom metaclasses overriding __new__: can return any class object
   • Otherwise: type's __new__ generates and returns the class object

2. Calls the metaclass's __init__ with the __new__-returned class object and definition info:

   • Custom metaclasses overriding __init__: can process the class object arbitrarily (unless slots mode fixes memory layout)
   • Otherwise: type's __init__ (empty method) is called

When instantiating using class objects as factory functions, the metaclass's __call__ is invoked:

1. Custom metaclasses overriding __call__: can return any object

2. Otherwise: type's __call__ executes the default instantiation flow:

   • Call class's __new__ to construct base object
   • Pass base object and parameters to class's __init__

6. Metaclass Instance Methods

Since a metaclass's instances are the classes using it as meta, methods defined in the metaclass become class methods for those classes.

Example:

class PointMeta(type):
    def __new__(cls, name, bases, namespace, **kwds):
        namespace["x"] = 0
        namespace["y"] = 0
        return super().__new__(cls, name, bases, namespace)
    
    def parse(cls, s):
        x_str, y_str = s.split(",")
        point = cls()
        point.x = int(x_str)
        point.y = int(y_str)
        return point

class Point(metaclass=PointMeta):
    pass

p = Point.parse("1,2")
assert p.x == 1
assert p.y == 2

Here, parse is an instance method of PointMeta, but for Point (which uses PointMeta as metaclass), parse becomes a class method.

7. Determining Instance Types

How do isinstance or type functions determine an object's class? Every object corresponds to memory space(s), and all information—including class membership—derives from this memory layout.

7.1 Non-Slots Mode: Dynamic Dictionary Layout (Default)

Python's most common mode prioritizes flexibility with this memory structure:

• PyObject Header: Contains reference count and pointer to type object
• dict Pointer: Points to an actual Python dictionary object
• weakref Pointer: Supports weak references

Advantages: Dynamic—attributes stored in __dict__ dictionary, allowing runtime addition of any member.

Disadvantages: High memory overhead (dictionary reserves space to reduce collisions), slower access due to hashing for each member lookup.

7.2 Slots Mode

Uses compact array layout. Defining __slots__ = ('a', 'b') structures object memory as:

• PyObject Header: Reference count and type pointer
• Fixed Offset Attribute Slots: Direct memory positions for a and b (storing pointers to actual objects)

No __dict__ (unless explicitly included in slots). This static-compiled-language-style layout means instances can only have slots-defined attributes; attempting to add new attributes raises AttributeError.

Benefits: Extremely compact memory layout eliminates entire dictionary object overhead. With millions of small objects, memory usage typically reduces 40%-70%. Attribute access becomes base address + fixed offset direct memory addressing, eliminating hash computation and improving performance.

Determining Instance Type

Regardless of memory layout, every object has a PyObject Header containing a pointer to the class object—this is the basis for type determination. During instantiation, who writes this pointer?

Python always calls object's __new__ for class instantiation. It calculates required memory size based on the specified type, allocates matching memory, and writes the class object's address into the PyObject Header's type pointer.

class object:
    def __new__(cls) -> Self:
        # Allocates memory and writes type pointer to PyObject Header
        ...

Clarification: While non-slots mode is "dynamic," this refers to the __dict__-pointed dictionary. The object's base memory contains only PyObject Header and two additional pointers.

Conclusion

Python's metaclass-based metamodel represents a sophisticated yet elegant design. Understanding the relationship between meta and instance, the role of type as the ultimate metaclass, and the instantiation flow provides deep insight into Python's object model.

This knowledge empowers developers to leverage metaclasses effectively for advanced metaprogramming scenarios while appreciating the consistency underlying Python's dynamic nature.