While Python has emerged as the predominant language in the AI and machine learning domain, developers coming from strongly-typed languages like C# often find numerous aspects of Python's design worthy of critique. However, there exists one absolute highlight in Python's architecture that stands out as a brilliant design decision: the meta-model based on metaclasses. This comprehensive exploration delves into the intricate workings of Python's meta-model, revealing the elegant mechanisms that power its type system.

1. The Metaverse Analogy: Understanding Meta-Concepts

To grasp the concept of meta-models, let's draw an analogy with the metaverse—a concept that captured global imagination in recent years. From a "creation" perspective, the metaverse defines the laws governing a universe (our reality), where the universe itself is an instance constructed from metadata. The relationship between "meta" and "instance" isn't absolute; the metaverse is itself a universe, and its laws are defined by a higher-level "meta-metaverse," making the metaverse an instance of this meta-metaverse.

This abstraction can continue indefinitely until we reach an ultimate尽头 where we cannot find the source of laws. Some refer to this as the "Creator," while Laozi called it "Dao" (the Way). We might call this the "Genesis Universe." Since we cannot find a meta for the Genesis Universe, we treat it as its own meta, forming a self-consistent closed loop. If we view the universe as an instance of its metaverse, then the Genesis Universe's instances include itself.

The meta can be simpler than the instance—as the saying goes, "the Great Dao is simple." The principle "Dao begets One, One begets Two, Two begets Three, Three begets all things" illustrates this perfectly. Conversely, the meta can also be more complex than the instance, containing intricate laws from which we select partial rules to construct instances. Readers familiar with "Perfect World" will recognize that the "Lower Realm's Eight Domains," serving as an exile ground for prisoners, represents a "dimension-reduced universe" constructed with the "Upper Realm" as its metaverse. Due to incomplete laws, cultivation in the Lower Realm can only reach the "Venerable Realm" and cannot achieve godhood.

Python's "metaverse" operates on similar principles. We use meta to define rules for instances and serve as a factory for creating instances. Thus, a class serves as the meta for its instances. We can use metaclasses to create classes, meaning conventional classes are instances of metaclasses, and metaclasses are the meta of conventional classes. In this sense, "class-meta" might be a more appropriate term than "metaclass," but metaclass not only expresses "the meta of class" but also embodies the concept that metaclasses themselves are classes. Since metaclasses are classes, they naturally can have their own metaclasses. Therefore, class is meta, and meta is also class—just as "the metaverse is also a universe, and a universe can serve as a metaverse."

Python's Genesis Universe is type. Our classes are constructed by it, making type the metaclass of classes. Since type is a metaclass, it possesses class attributes. Due to its transcendent position in the Genesis Universe, type can be viewed as an instance of itself.

2. Defining Metaclasses: The Creation Process

By default, when we use a class to create an instance, the underlying process follows these steps:

  1. Object Creation: Pass the class and parameters (including any keyword arguments) to the class's __new__ method to create a base object
  2. Initialization: Pass this base object and parameters to the class's __init__ method for initialization
  3. Return: Return the initialized object

Consider the following Foobar class definition. Two methods of creating instances (foobar1 and foobar2) are equivalent:

from typing_extensions import Self
from typing import Any

class Foobar:
    def __new__(cls, *args: Any, **kwargs: Any) -> Self:
        return super().__new__(cls)
    
    def __init__(self, foo: int, bar: int) -> None:
        self.foo = foo
        self.bar = bar
    
    def __eq__(self, value: object) -> bool:
        if not isinstance(value, Foobar):
            return NotImplemented
        return self.foo == value.foo and self.bar == value.bar

# Method 1: Standard instantiation
foobar1 = Foobar(foo=111, bar=222)

# Method 2: Explicit __new__ and __init__ calls
foobar2 = Foobar.__new__(Foobar, foo=111, bar=222)
Foobar.__init__(foobar2, foo=111, bar=222)

assert foobar1 == foobar2

Important Note: Although __new__ appears to be a class method based on its definition, it is本质上 a static method. It lacks the standard @classmethod decorator, and when called, the first parameter must be explicitly specified.

Similarly, a metaclass, being a class itself, can define __new__ and __init__ methods. While these methods in a regular class serve to initialize its instances, in a metaclass, their mission is the same—the only difference is that the metaclass's instances are the classes that use it as their metaclass.

When the Python interpreter encounters a class definition with a specified metaclass, it employs a similar approach to create the class:

  1. Class Object Creation: Call the metaclass's __new__ method with fixed parameters:

    • The metaclass itself
    • Class name
    • Class members (represented as a dictionary)
    • Keyword arguments specified during class definition

  2. Class Initialization: Call the metaclass's __init__ method to initialize the base class object created above:

    • self is the class object
    • Remaining parameters include class name, class members dictionary, and keyword arguments

Consider the following metaclass Meta as an example. We define __new__ and __init__ methods that print all output parameters. In __new__, we add keyword arguments to the namespaces dictionary representing class members, then call the base class (type)'s __new__ method to create the base class object. In __init__, we define a repr function that outputs as a string and set it as the class's __repr__ method:

from typing import Any

class Meta(type):
    def __new__(cls, name: str, bases: tuple[type, ...], namespaces: dict[str, Any], /, **kwds: Any):
        print(f"""
__new__
 cls: {cls}
 name: {name}
 bases: {bases}
 namespaces: {namespaces}
 kwds: {kwds}
""")
        for key, value in kwds.items():
            namespaces[key] = value
        return super().__new__(cls, name, bases, namespaces)
    
    def __init__(self, name: str, bases: tuple[type, ...], dict: dict[str, Any], /, **kwds: Any) -> None:
        print(f"""
__init__
 cls: {self}
 name: {name}
 bases: {bases}
 dict: {dict}
 kwds: {kwds}
""")
        def repr(self) -> str:
            express = ", ".join(f"{key}={value!r}" for key, value in kwds.items())
            return f"({express.strip()})"
        setattr(self, "__repr__", repr)

class Foo:
    ...

class Bar(Foo, metaclass=Meta, x=-1, y=-1):
    ...

print(Bar())

In this example, Meta serves as the metaclass for class Bar, while Foo is Bar's base class. When defining Bar, we specify two keyword arguments. According to Meta's __new__ method definition, they become field members of the type. After creating the Bar object and passing it to the print method, the program produces the following output:

__new__
 cls: <class '__main__.Meta'>
 name: Bar
 bases: (<class '__main__.Foo'>,)
 namespaces: {'__module__': '__main__', '__qualname__': 'Bar', '__firstlineno__': 33, '__static_attributes__': ()}
 kwds: {'x': -1, 'y': -1}

__init__
 cls: <class '__main__.Bar'>
 name: Bar
 bases: (<class '__main__.Foo'>,)
 dict: {'__module__': '__main__', '__qualname__': 'Bar', '__firstlineno__': 33, '__static_attributes__': (), 'x': -1, 'y': -1}
 kwds: {'x': -1, 'y': -1}

(x=-1, y=-1)

This output demonstrates how the metaclass intercepts class creation, modifies the namespace, and injects custom behavior into the resulting class.

3. Instances Are Created by Metaclasses: The Hidden Truth

The extensive discussion above aimed to illustrate the class instance creation flow: first call __new__ to build a base object, then pass it to __init__ for initialization. However, this is merely the surface appearance, not the essence. The fundamental truth is: instances of a class are created by its metaclass.

When we speak of "using a metaclass object to construct," we essentially treat the metaclass as a factory function for instance construction. In Python's world, everything is an object—including functions. A function is an instance of the function class, and the function (or callable) class is a class possessing a __call__ method (either self-defined or inherited from a base class). Calling a function is本质上 calling the __call__ method of the function object.

When invoking a metaclass to construct an instance in function form, it means the instance is created through the __call__ method defined in the metaclass. The following demonstration fully illustrates this point:

from typing import Any

class Foo:
    ...

class Meta(type):
    def __call__(self, *args: Any, **kwds: Any) -> Any:
        assert self is Bar
        assert type(self) is Meta
        assert args == ("111", "222")
        assert kwds == {"c": "333", "d": "444"}
        return Foo()

class Bar(metaclass=Meta):
    def __init__(self, a: str, b: str, **kwargs) -> None:
        self.x = a
        self.y = b
        self.kwargs = kwargs

assert isinstance(Bar("111", "222", c="333", d="444"), Foo)

As a method, its first parameter is always the subject object of the method call (for class methods, the subject is the class; for instance methods, the subject is the class instance). From the assertions in the __call__ method, we can see that the subject object calling this method is the Bar class object itself. The type(self) returns the class, which is naturally the metaclass Meta that defines this class. The positional and keyword arguments injected into Bar("111", "222", c="333", d="444") are assigned to args and kwds in tuple and dictionary form, respectively.

So why does calling Foo() return a Foo object for ordinary classes like Foo? The same rule still applies: although Foo doesn't specify a concrete metaclass, it means type serves as the fallback metaclass. The instance returned by Foo() is actually created by the __call__ method defined in type, which constructs objects as follows:

class type:
    def __call__(self, *args: Any, **kwds: Any) -> Any:
        instance = self.__new__(self, *args, **kwds)  # type: ignore
        self.__init__(instance, *args, **kwds)
        return instance

This method creates objects through the following process:

  1. First call the class's __new__ to build a base object
  2. Then pass that object as a parameter to the class's __init__ method for initialization, finally returning the object

We don't need to worry about whether a class defines __new__ and __init__, because the ultimate base class object provides the fallback:

  • A defined parameterless __new__ allocates memory as an empty created object
  • A defined parameterless __init__ method does nothing

4. The Ultimate Metaclass: type

The __call__ method and the type class are far more complex than the instance construction logic described above. Let's explore the deeper mechanisms.

4.1 The type.__new__ Method

First, let's examine the logic of the __new__ method defined in the type class, which is used to construct a class object. The __new__ method of our custom metaclass Meta ultimately calls this method (return super().__new__(cls)). Here is the method signature:

class type:
    def __new__(
        cls: type[Self],
        name: str,
        bases: tuple[type, ...],
        namespace: dict[str, Any], /,
        **kwds: Any
    ) -> Self: ...

The five parameters are:

  • cls: The metaclass constructing the class
  • name: Used to name the ultimately constructed class
  • bases: Base classes for the constructed type
  • namespaces: Type members
  • kwds: Additional keyword arguments

If the cls parameter is the type class object itself, the constructed result is a conventional type, named by the name parameter, with bases as base classes, and possessing type members defined by namespaces. The keyword arguments specified by the kwds parameter are meaningless in this case:

class Foo:
    x = -1

cls = type.__new__(type, "Bar", (Foo,), {"y": -1})
assert cls.__name__ == "Bar"
assert cls.__bases__ == (Foo,)
assert type(cls) is type

bar = cls()
assert bar.x == -1
assert bar.y == -1

Things become more interesting when the cls parameter specifies a custom metaclass. Since both the metaclass and type.__new__ method can create types, conflicts arise. Clearly, the latter has higher priority (after all, we explicitly called __new__), so the custom metaclass's __new__ and __init__ methods won't take effect. The following demonstration proves this:

class Baz:
    ...

log = []

class Meta(type):
    def __new__(cls, name: str, bases, namespaces, /, **kwds):
        log.append(f"Meta.__new__ is called")
        return Baz
    
    def __init__(self, *args, **kwargs):
        log.append(f"Meta.__init__ is called")
        self.z = -1

class Foo:
    x = -1

cls = type.__new__(Meta, "Bar", (Foo,), {"y": -1})

assert len(log) == 0
assert cls.__name__ == "Bar"
assert cls.__bases__ == (Foo,)
assert type(cls) is Meta

bar = cls()
assert bar.x == -1
assert bar.y == -1
assert not hasattr(bar, "z")

Although the __new__ and __init__ methods defined by the metaclass don't take effect, the specified metaclass is indeed set as the generating type's metaclass (assert type(cls) is Meta). If we override the __call__ method in the metaclass, according to the rules introduced above: when we use the generated class for instantiation, the method called behind the scenes is this one, as reflected in the following demonstration:

class Baz:
    ...

class Meta(type):
    def __call__(self, *args, **kwargs):
        return Baz()

class Foo:
    x = -1

cls = type.__new__(Meta, "Bar", (Foo,), {"y": -1})
assert isinstance(cls(), Baz)

4.2 The type Function

As the ultimate metaclass, type is also a class. As mentioned above: when we treat a class as an executable function, the metaclass's __call__ method is called behind the scenes. Since type's metaclass is itself, calling the type function is本质上 calling type's __call__ method.

We know the logic of the type function is as follows:

  1. Single Object Input: If a single object is passed, it returns the class of the object. Specifically:

    • If a regular object is specified, return that object's class
    • If a class is specified, return its metaclass
    • If no metaclass is explicitly specified, the fallback metaclass type is naturally returned

This rule is well demonstrated in the following code:

class Meta(type):
    pass

class Foobar(metaclass=Meta):
    pass

assert type(Foobar()) is Foobar
assert type(Foobar) is Meta
assert type(type) is type
  1. Multiple Parameters: If the single parameter requirement isn't met, the type function ultimately creates a class, requiring input parameters in order: class name, base classes, class members, keywords (adding type in front, exactly matching __new__ method parameters):
class Base():
    foo = -1

cls = type("Foobar", (Base,), {"bar": -1})
assert cls.__name__ == "Foobar"
assert cls.__bases__ == (Base,)
assert cls.bar == -1  # type: ignore

instance = cls()
assert instance.foo == -1
assert instance.bar == -1  # type: ignore

Therefore, type's __call__ determines whether to return the class of the specified object or create a new class based on the parameter format. The following definition更接近 the actual implementation:

class type:
    def __call__(self, *args, **kwargs):
        # Single parameter: return the parameter's type
        if len(args) == 1 and not kwargs:
            obj = args[0]
            return getattr(obj, "__class__", type(obj))
        
        # Multiple parameters: create a new class
        instance = self.__new__(self, *args, **kwargs)
        self.__init__(instance, *args, **kwargs)
        return instance

5. Reorganizing Class Generation and Instantiation: A Comprehensive Summary

Let me now provide a comprehensive summary of class generation and class-based instantiation:

Class Generation Process

For a class code snippet written using the class keyword, the Python interpreter constructs this class as follows:

  1. Extract the Metaclass: If not explicitly specified, the type class serves as the metaclass

  2. Call Metaclass new: Pass the metaclass and class definition information (class name, base classes, all members defined for the class, and keyword arguments) as parameters to the metaclass's __new__ method:

    • For custom metaclasses, if it (or its custom base class) overrides the __new__ method, it can theoretically return any class object
    • If no metaclass is explicitly specified, or the specified custom metaclass (including its custom base class) doesn't override __new__, then type's __new__ method is called. Following the logic above, it strictly generates and returns the corresponding class object in the form we defined

  3. Call Metaclass init: Pass the class object returned by __new__, along with class definition information, as parameters to the metaclass's __init__ method:

    • If the specified custom metaclass (or its custom base class) overrides the __init__ method, and if not using slots mode, it can theoretically perform arbitrary processing on the passed class object
    • Once the class object is created based on slots mode, since the class's memory layout is fixed, class members cannot be added or deleted
    • If no metaclass is explicitly specified, or the specified custom metaclass (including its custom base class) doesn't override __init__, then type's __init__ method is called—this method is empty and performs no operations

Class Instantiation Process

When we use a class object as a factory function for instantiation, the metaclass's __call__ method is invoked, where the first parameter is the class object serving as the factory function:

  1. Custom call: If the specified custom metaclass (or its custom base class) overrides the __call__ method, it can theoretically return any object

  2. Default call: If no metaclass is explicitly specified, or the specified custom metaclass (including its custom base class) doesn't override __call__, then type's __call__ method is called, reflecting the default instantiation flow:

    • Call the class's __new__ method to build a base object:

      • If the class (or its custom base class) overrides __new__, it can theoretically return any object
      • Otherwise, the __new__ method inherited from the object class is used to create this base object

    • Pass the base object built by __new__, along with specified parameters, to the class's __init__ method:

      • If the class (or its custom base class) overrides __init__, it can theoretically perform any processing on the provided base object
      • Otherwise, the __init__ inherited from object is called, but it does nothing

6. Metaclass Instance Methods: Becoming Class Methods

Since a metaclass's instances are the classes that use it as their metaclass, for classes created by it, instance methods defined in the metaclass become their class methods. Consider the following PointMeta metaclass as an example: parse is its instance method. However, for Point, which uses it as a metaclass, parse becomes its class method:

class PointMeta(type):
    def __new__(cls, name, bases, namespace, **kwds):
        namespace["x"] = 0
        namespace["y"] = 0
        return super().__new__(cls, name, bases, namespace)
    
    def parse(cls, s):
        x_str, y_str = s.split(",")
        point = cls()
        point.x = int(x_str)
        point.y = int(y_str)
        return point

class Point(metaclass=PointMeta):
    pass

p = Point.parse("1,2")
assert p.x == 1
assert p.y == 2

This elegant mechanism allows metaclasses to inject class-level methods into the classes they create, enabling powerful meta-programming patterns.

7. Determining Instance Type: The Memory Layout Question

Have you ever wondered how isinstance or type functions determine the class to which a specified object belongs? Some might say, "doesn't every object have a __class__ field?" That's correct, but where does this field come from?

Regardless of the programming language, an object always corresponds to one (continuous) or multiple (discontinuous) memory spaces. All information provided by the object originates from this memory, including the class to which the object belongs. Therefore, understanding an object's memory layout reveals all information about the object. For a regular object (not a class object—class object layouts are much more complex), its memory layout depends on whether it uses slots mode.

7.1 Non-Slots Mode: Dynamic Dictionary Layout (Default)

This is Python's most commonly used mode, with flexibility at its core. Memory is laid out using the following structure:

  • PyObject Header: Contains reference count and a pointer to the type object
  • dict Pointer: Points to a real Python dictionary object
  • weakref Pointer: Used to support weak references

Dynamic capability is the greatest advantage of this layout. Since attributes are not stored in the instance itself but in the dictionary pointed to by __dict__, we can add arbitrary members at any time. However, the trade-off is high memory overhead. Since dictionaries reserve space to reduce conflicts, each instance must additionally maintain a hash table object. Each data member lookup requires hashing, affecting access speed.

7.2 Slots Mode: Compact Array Layout

This mode employs a compact array layout. When we define __slots__ = ('a', 'b'), Python uses the following structure to layout object memory:

  • PyObject Header: Reference count and type pointer
  • Fixed Offset Attribute Slots: Directly reserve positions for a and b in memory (storing pointers to specific objects)

There is no __dict__ (unless you explicitly add 'dict' in slots). This is the memory layout mode adopted by statically compiled languages, resulting in instances being able to possess only members defined in slots. Attempting to add new attributes raises an AttributeError.

Although dynamic capability is lost, the extremely compact memory layout eliminates the overhead of the entire dictionary object. When owning millions of small objects, memory usage can typically be reduced by 40%-70%. Attribute access becomes direct memory addressing (base address + fixed offset) without hash calculations, significantly improving performance.

Returning to the Title Question: How to Determine Instance Type?

The answer is simple: regardless of which memory layout is used, each has a PyObject Header containing a pointer to the class object. This is the basis for determining object type.

So in the instantiation flow described above, who writes this pointer? Python's instantiation for a certain class always calls object's __new__ method, which calculates the memory size required for the object based on the specified type and allocates a matching memory segment. It then writes the class object's address into the type pointer in the PyObject Header.

Some might ask, "Didn't we just say that object memory in non-slots mode is dynamically allocated?" While that's true, the "dynamic" here refers to the dictionary pointed to by __dict__. The memory here only includes the PyObject Header and two additional pointers.

class object:
    def __new__(cls) -> Self:
        # Allocates memory and sets type pointer in PyObject Header
        ...

Conclusion: The Elegance of Python's Meta-Model

Python's meta-model represents a masterful balance between flexibility and structure. The metaclass mechanism provides developers with unprecedented control over class creation and behavior, enabling sophisticated meta-programming patterns while maintaining an elegant and consistent type system.

Understanding this meta-model is crucial for:

  • Advanced Framework Development: Building ORMs, serialization libraries, and validation frameworks
  • API Design: Creating intuitive and expressive interfaces
  • Performance Optimization: Leveraging slots and custom metaclasses for memory efficiency
  • Code Generation: Automating repetitive class definitions with meta-programming

The journey from the Genesis Universe (type) through metaclasses to regular classes and their instances reveals the profound depth of Python's object model—a testament to the language's design philosophy of providing powerful abstractions while maintaining simplicity and consistency.