本文探讨了John Wentworth和David Lorell提出的“中介变量决定冗余”定理，并将其从图模型语言转换为信息论语言。新定理假设随机变量集合中存在中介变量Λ和冗余变量Λ′，通过限定条件下的互信息和熵约束，推导出Λ′的熵上界。与原定理相比，信息论版本在表述上更简洁，但两者通过d分离准则和互信息零等价性保持等价性。该工作有助于深入理解变量间信息依赖关系，为因果结构分析提供新视角。

📊 新定理在信息论框架下重新表述了中介变量与冗余变量的关系，假设变量集合包含Λ（中介）和Λ′（冗余），通过限定条件下的互信息和熵约束推导出Λ′的熵上界。

🔹 定理的核心条件包括：Λ对A和B子集的信息传递限制（I(XA:XB|Λ)≤ϵmed），冗余变量对子集的独立性限制（H(Λ′|XA)≤ϵred），以及信息论不等式的应用。

🔄 证明过程通过条件熵和互信息的分解，结合Redundancy（冗余）和Mediation（中介）假设，最终推导出H(Λ′|Λ)≤ϵmed+2ϵred的结论，展示了变量间信息依赖的量化关系。

🌐 该工作将原始图模型表述转换为信息论语言，利用d分离准则和互信息零等价性证明两者等价性，为变量依赖分析提供新工具。

🔬 通过将随机变量替换为其具体实例，该推导可扩展至Kolmogorov复杂度和算法互信息，适用于更广泛的概率模型和因果结构分析。

Published on October 26, 2025 8:33 PM GMT

This post is a comment on Natural Latents: Latent Variables Stable Across Ontologies by John Wentworth and David Lorell. It assumes some familiarity with that work and does not attempt to explain it. Instead, I present an alternative proof that was developed as an exercise to aid my own understanding. While the original theorem and proof are written in the language of graphical models, mine instead uses the language of information theory. My proof has the advantage of being algebraically succinct, while theirs has the advantage of developing the machinery to work directly with causal structures. Very often, seeing multiple explanations of a fact helps us understand it, so I hope someone finds this post useful.

Specifically, we are concerned with their Theorem 1 (Mediator Determines Redund): both the older Iliad 1 version for stochastic latents, and the newer arXiv version for deterministic latents. I will translate each theorem into the language of information theory: Wentworth & Lorell's assumptions will imply mine, while their conclusions will be equivalent to mine. The equivalences follow from the d-separation criterion and the fact that independence is equivalent to zero mutual information.

In our version of the new theorem, $\Lambda$ is a mediator between subsets A and B of the data, meaning that it contains essentially all of the information in common between A and B, whereas ${\Lambda }^{}$ is a redund between A and B, meaning it essentially only contains information that is in common between A and B.^[1]

New Theorem 1 (deterministic latents)

Let A,B be disjoint subsets of {1,...,n}.

Suppose the random variables $X_1,\dots X_n,\Lambda ,{\Lambda }^{}$ satisfy the following:

$\Lambda$ Mediation: $I(X_A:X_B\mid \Lambda )\le {ϵ}_{med}$,

${\Lambda }^{}$ Redundancy: $H({\Lambda }^{}\mid X_A)\le {ϵ}_{red}$ and $H({\Lambda }^{}\mid X_B)\le {ϵ}_{red}$.

Then, $H({\Lambda }^{}\mid \Lambda )\le {ϵ}_{med}+2{ϵ}_{red}$.

Proof

$H({\Lambda }^{}\mid \Lambda )$

$=H({\Lambda }^{}\mid X_B,\Lambda )+I({\Lambda }^{}$            by definition of conditional mutual information,

$\le H({\Lambda }^{}\mid X_B)+I(X_A:X_B\mid \Lambda )+H({\Lambda }^{}$             by information theory inequalities,

$\le {ϵ}_{med}+2{ϵ}_{red}$                                                                       by Redundancy and Mediation.

Old Theorem 1 (stochastic latents)

Suppose the random variables $X_1,\dots X_n,\Lambda ,{\Lambda }^{}$ satisfy the following:

Independent Latents: $I(\Lambda :{\Lambda }^{}\mid X)\le {ϵ}_{ind}$,

$\Lambda$ Mediation: $I(X_j:X_{-j}\mid \Lambda )\le {ϵ}_{med}$ for all j,

${\Lambda }^{}$ Redundancy: $I({\Lambda }^{}:X_j\mid X_{-j})\le {ϵ}_{red}$ for all j.

Then, $I({\Lambda }^{}:X\mid \Lambda )\le n({ϵ}_{ind}+{ϵ}_{med}+{ϵ}_{red})$.

Proof

First, we have

$I({\Lambda }^{}:X_j\mid X_{-j})-I({\Lambda }^{}$

$=I({\Lambda }^{}:X_j:\Lambda \mid X_{-j})$                                  by definition of 3-way interaction information,

$=I({\Lambda }^{}:\Lambda :X_j\mid X_{-j})$                                  by symmetry of 3-way interaction information,

$=I({\Lambda }^{}:\Lambda \mid X_{-j})-I({\Lambda }^{}$

$\ge -I({\Lambda }^{}:\Lambda \mid X_j,X_{-j})$

$\ge -{ϵ}_{ind}$                                                            by Independent Latents.

Therefore,

$I({\Lambda }^{}:X_j\mid \Lambda )$

$\le I\left(\right({\Lambda }^{},X_{-j}):X_j\mid \Lambda )$

$=I(X_{-j}:X_j\mid \Lambda )+I({\Lambda }^{}:X_j\mid \Lambda ,X_{-j})$           by mutual information chain rule,

$\le I(X_{-j}:X_j\mid \Lambda )+I({\Lambda }^{}:X_j\mid X_{-j})+{ϵ}_{ind}$     by the above derivation,

$\le {ϵ}_{ind}+{ϵ}_{med}+{ϵ}_{red}$                                                by Mediation and Redundancy.

The result now follows by summing over all j=1,...,n.

1. ^{^}

   Since probabilistic models are often only defined in terms of a latent structure, you might find it philosophically suspect to impose a joint distribution on all variables including the latents. If so, feel free to replace the random variables with their specific instantiations: the derivations go through almost identically with Kolmogorov complexity and algorithmic mutual information replacing the Shannon entropy and mutual information, respectively.

Discuss