The Shadow of Compression
On symbolic and statistical learning as two directions toward the same fixed point
Tldr:
Learning is compression. That is the thesis, and everything else in this essay is an attempt to take it seriously.
When an expert writes down a system specification, they are compressing their accumulated understanding of a domain into a form that can be checked, shared, and executed. When a neural network generalizes beyond its training set, it is compressing a pile of examples into a rule that produces the right answer on examples it has never seen. When a physicist writes F = ma, they are compressing centuries of mechanical observation into three symbols. These look like different activities. They are not. They are the same activity performed by different kinds of searchers, and once you see that, a surprising amount of the debate about symbolic versus statistical AI, about Models based systems engineering versus learned models, about whether large language models really “understand” anything, starts to dissolve into a more tractable question: what compression are you using, at what scale is it valid, and how would you know if it stopped being valid?
That last question is where the essay ends up and where it gets hard. Every compression works by throwing things away. A good compression throws away the things that do not matter. But the things that do not matter today can start mattering tomorrow, and when they do, the model that was built to ignore them is the last place the warning will show up. This is not a theoretical concern. It is the shape of most modeling failures in complex systems — economic models that worked until they did not, safety cases that held until the regime shifted, forecasts that were correct about the world they were built for and wrong about the world that actually arrived. The discipline this essay is trying to name is the discipline of knowing which world your compression is for. The essay’s Addendum is pointing at that gap, and it’s arguably the most important question for anyone building systems that need to keep working when the world changes.
1. The shadow and what it hides
Conservation laws are the shadow cast by symmetry.
The line is a folk distillation of Noether’s theorem, and it captures something that took physics a long time to see clearly. Energy, momentum, angular momentum — the quantities we once treated as fundamental features of the world — turn out to be consequences of something deeper. They are what you observe when a system is indifferent to certain transformations: time translation, spatial translation, rotation. The symmetry is the cause; the conservation law is the visible effect. Shadow and object.
This reframing matters because it changes what counts as an explanation. Before Noether, “energy is conserved” was a law you memorized. After Noether, it is a consequence you derive from a more basic fact about the structure of physical action. The conservation law did not go away. It was relocated — moved from the foundation to the surface, from the thing you assume to the thing you predict.
There is a similar move available in how we think about learning. When a student masters a domain, or a neural network generalizes beyond its training set, or an engineer writes down a system model that actually survives contact with reality, we tend to describe what happened in terms of rules discovered. The student learned the rule. The network found the pattern. The engineer captured the dynamics. But the rule, like the conservation law, may be the shadow of something deeper. What is actually happening — in all three cases — is the identification of a minimal sufficient compression of a domain, and the rule we can articulate afterward is the visible trace of that compression.
If that is right, then a great deal of apparently opposed work in AI, in systems engineering, and in the philosophy of modeling is really the same work approached from different directions. The essay that follows is an attempt to say what that work is, and what its shadow hides.
2. Compression as the definition of understanding
Algorithmic information theory, the tradition founded independently by Kolmogorov, Solomonoff, and Chaitin in the 1960s, gives us a way to make this precise. The central object is the Kolmogorov complexity of a string, written K(s): the length, in bits, of the shortest program that produces s as output on some fixed universal computer. A regular string like `010101…01` has low complexity — a short loop will print it. A random string of the same length has high complexity — the shortest program that produces it is essentially the string itself. Randomness, on this view, is incompressibility.
The move that matters for us is the decomposition of any candidate model P of a dataset D into two parts:
P = R + E
where R is the rule component — the regularity, the pattern, the part that compresses — and E is the residual, the exception, the stuff R does not explain. The total description length of the model is |R| + |E|, and the minimum description length principle says: among all models that reproduce D, prefer the one that minimizes |R| + |E|. Simpler is better, but only up to the point where further simplification forces you to pay for too many exceptions.
This gives us the first rigorous distinction between rote learning and understanding. A pure memorizer has |R| = 0 and |E| = |D|; it has compressed nothing. A perfect rule has |E| = 0 and |R| much smaller than |D|; it has compressed everything. A good model lives somewhere in between, with small R capturing the dominant structure and small E absorbing the noise. Understanding, in this framing, is not a binary state. It is a position along a compression gradient, and the quality of understanding is the degree to which the total description has been shortened relative to the raw data.
Two caveats matter. First, Kolmogorov complexity is incomputable in general — there is no algorithm that takes a string and returns its shortest program. Everything we do in practice is an approximation. Regularization in neural networks approximates it. Bayesian priors over program length approximate it. Human intuitions about elegance and parsimony approximate it. The approximation may be good or bad, and we rarely know which.
Second, the decomposition R + E is not unique. For a given dataset there are many ways to split the model between rule and residual, and the split that minimizes total description length depends on which programming language you count bits in. This is not a fatal problem — different reasonable choices converge up to a constant — but it tells us something important. The question “what is the rule here?” does not have a single answer independent of the representational language you are using to ask it.
That second point is where the next layer of the argument enters.
3. Harmonics and scale: the renormalization frame
The representational language question is exactly the question the re-normalization group was built to answer. Renormalization Group is a technique from statistical physics for dealing with systems that have structure at many scales simultaneously — a fluid, a magnet near its critical point, a quantum field. The trick is to systematically integrate out the short-distance degrees of freedom and ask what the system looks like at longer distances. You coarse-grain. You smear out details below some resolution and see what survives.
What survives, in the Renormalization Group picture, falls into two categories. *Relevant* operators grow under coarse-graining; they matter more at long distances than at short ones, and they control the large-scale behavior of the system. *Irrelevant* operators decay; they may be important at microscopic scales but they wash out as you zoom out, and at long distances they contribute only corrections. The universality classes of physics — the reason wildly different microscopic systems can exhibit identical critical behavior — are the consequence of this filtering. Many possible microscopic theories flow to the same macroscopic fixed point because their differences were all in irrelevant operators.
Translate this into the language of compression. A function — the input-output behavior of a system — can be decomposed into components at different scales. Call them harmonics, borrowing the intuition from Fourier analysis: each harmonic captures structure at a particular characteristic scale, and the full function is some combination of them. Now the Renormalization Group and Algorithmic information theory pictures align. Relevant harmonics are those with low Kolmogorov complexity relative to the variance they explain — they are cheap to describe and they carry a lot of the signal. Irrelevant harmonics are expensive to describe relative to their explanatory power, and the minimum description length principle tells us to absorb them into the residual E. Coarse-graining is compression. The fixed point of Renormalization Group flow is the minimal sufficient representation.
Here is the critical point, and it is the one most often missed when people apply these ideas loosely. *Relevance is scale-dependent.* A harmonic that is irrelevant at one scale can be the dominant signal at another. Temperature fluctuations are noise at the thermodynamic scale and structure at the molecular scale. The question is never “what are the patterns in this data?” in the abstract. It is always “what are the patterns that matter at the scale at which I need to act?”
This has a consequence we will come back to. When someone builds a model — any model, whether a set of differential equations or a trained network or a systems diagram — they are not just compressing the data. They are committing to a scale. The model’s claim on reality is only valid at the scale where its compression is appropriate. Applied at a finer scale, the relevant and irrelevant designations flip, and the model’s “rules” become wrong in specific, diagnosable ways.
The work of understanding a domain, then, has two inseparable parts. You have to find the relevant harmonics, and you have to find them at the right scale. Get either wrong and the compression fails.
4. Two directions to the same fixed point
With this machinery in place the central claim of the essay can be stated cleanly. Symbolic and statistical approaches to learning are not opposed paradigms. They are two search strategies operating on the same space — the space of possible compressions of a domain — and they converge, when the task’s structure is sufficiently constrained, on the same fixed points. What distinguishes them is the direction of approach.
The convergence claim requires a condition that is easy to state and easy to forget. When a task’s symmetry group fully determines the irreducible representations — when there is, mathematically, only one way to decompose the signal at a given scale — any sufficient search must find the same compression, because there is only one compression to find. Modular arithmetic on a cyclic group is like this: the discrete Fourier basis is forced by the group structure, and no alternative decomposition achieves the same efficiency. Much of physics is like this, which is why physics-informed neural networks work. But not all domains have this property, and the essay will need to come back to the cases that do not.
The symbolic direction is top-down. An expert, through years of exposure to a domain, has already performed something like an Renormalization Group flow in their own cognition. Their intuitions about what matters and what can be ignored are the product of a long biological coarse-graining: many cases seen, many regularities noticed, many irrelevances forgotten. When they articulate rules — a set of equations, a type system, a requirements specification — they are committing their already-compressed representation to a formal language. The rule comes first, and the data is fit afterward. Parameter estimation, calibration, tuning: these are the residual operations, the fitting of E once R has been supplied by the expert.
A note on the stylization. The top-down / bottom-up distinction is a simplification. It is not a claim about how expert cognition works. The narrower claim is this: the symbolic methodology commits to a compressed representation first and fits data to it; the statistical methodology commits to data first and has a compression emerge under optimization pressure. The fixed point the two converge on is not a shared cognitive state but a shared formal compression — an equation, a weight configuration, a type system — that can be inspected and compared. That is the version the rest of the essay depends on.
Model-based systems engineering is the cleanest contemporary instance of this direction. In MBSE, the engineer begins by formalizing the system — its components, its interfaces, its behaviors, its constraints — as an explicit model. The model *is* the compressed representation. Requirements flow from it, tests flow from it, parameter values are chosen to make it match observed behavior. The whole discipline is built around the claim that a human expert, given enough care and the right formalism, can write down the relevant harmonics of a complex system before the system is built or measured in detail. When MBSE works, it is because the expert’s compression was good. When it fails, it is because the compression was not.
The statistical direction is bottom-up. A neural network, at initialization, has the capacity to represent an enormous space of functions but no commitment to any particular compression. Training proceeds in two phases, which we can now describe in the language we have built. In the first phase, the network memorizes: it stores the training data in a high-complexity representation that reproduces the inputs faithfully but compresses nothing. |R| is essentially zero; |E| is essentially |D|. The network has found *a* fit but not *the* compression.
The second phase is the one that took the field by surprise. Under sustained regularization pressure — weight decay being the canonical example, acting as a rough proxy for a Kolmogorov prior — the network continues to evolve long after it has perfectly fit the training data. Slowly, and then suddenly, it discovers a structurally different solution: a compressed representation in which the relevant harmonics of the task are encoded explicitly in the weights. This phenomenon, called *grokking* after its discovery in small transformers trained on modular arithmetic, is a phase transition from the memorizing solution to the compressing one. The clearest published case study concerns networks trained to compute a + b mod p. After grokking, the networks are found to have learned the discrete Fourier transform on the cyclic group — they represent each input as a combination of sinusoids at a handful of key frequencies, and they compute addition by applying trigonometric identities to those representations. The network has rediscovered harmonic analysis, not because it was told to, but because harmonic analysis is the minimum description length solution to arithmetic on a cyclic group, and the regularization pressure eventually selects for it.
Notice what has happened. The expert working in MBSE and the network undergoing grokking have arrived at the same kind of thing: a representation in which the relevant harmonics of the task are made explicit, and in which the residual is small. The expert got there by starting from the compressed side and projecting onto the data. The network got there by starting from the data and climbing, through gradient descent and regularization, toward the compressed side. The endpoints coincide, or would coincide, if both searches reached their optima. The symbolic and the statistical are not in opposition. They are reciprocal. They are two ways of approaching the same fixed point from opposite directions, and the fixed point itself is indifferent to which direction you came from.
But this convergence has a boundary that needs to be marked. The endpoints coincide when the task’s structure forces a unique compression — when the symmetry is rich enough that there is only one set of relevant harmonics to find. In such domains, the expert and the network are triangulating a fixed point that exists independently of either of them. In domains where the structure admits multiple valid decompositions — where several different sets of harmonics each compress the data comparably well — the two approaches may find different fixed points, both locally optimal, both compressing effectively, but not equivalent. The direction of approach becomes not merely a search strategy but a commitment to a particular way of carving the domain. This is a weaker claim than universal convergence, but it is a more interesting one: the choice between symbolic and statistical is not just a pragmatic decision about efficiency but a substantive decision about which compression you will live with.
5. The failure modes are diagnosable
If the two directions converge when they succeed, they also fail in mirror-image ways when they do not. Naming these failure modes precisely is where the framework earns its keep, because it turns vague complaints about “brittle rules” or “data-hungry models” into structural diagnoses.
The expert-driven approach fails when the expert’s biological Renormalization Group flow has converged to a local fixed point rather than the global one. The relevant harmonics they have internalized are real within the range of their experience, but they are not universal. A subtler harmonic that matters outside that range has been coarse-grained away, and the expert does not know it is missing because their cognitive apparatus was never tuned to notice it. When this happens, the formalized system — the MBSE model, the rule base, the governing equations — encodes the wrong R. No amount of subsequent parameter fitting can rescue it, because parameter fitting only adjusts E. The structural commitment has already been made. The model will perform well within the regime the expert’s intuition covered and will fail, often without warning, at its boundaries. The failure mode of the expert is to be *confidently wrong* in a way that looks locally right.
The statistical approach fails for a dual reason. Gradient descent is not guaranteed to find the minimum-complexity program. It finds a compression, but not necessarily the compression. The generalizing basin in parameter space may exist, but it may be small, hard to reach, or shielded from the memorizing basin by a ridge of high loss. Without the right architectural priors, the search is astronomical. We see this every time a modern model requires orders of magnitude more data or compute than intuition suggests it should, in order to rediscover a regularity that a domain expert could have written down in an afternoon. Worse, the network may settle into a compression that fits the training distribution well and fails, again often without warning, on inputs drawn from a slightly different distribution. It has found a fixed point, but not *the* fixed point — a compression that is valid at the scale of its training data and breaks down at other scales. The failure mode of the network is to be *expensively right about the wrong thing*.
These failures are duals of each other, and recognizing the duality is the first step toward not committing them. The expert fails by over-trusting their prior. The network fails by under-constraining its search. Both failures are, in the language of the previous section, failures to land at the minimal sufficient compression for the scale at which the model will actually be used. And both failures are invisible to the approach that caused them — the expert cannot see the harmonics they have coarse-grained away, and the network cannot see that a simpler solution exists in a basin it has not explored.
This is what the shadow metaphor hides, if we take it too literally. A shadow tells you an object is there, but it tells you nothing about the objects you do not see because they happen to be aligned with your light source. Both the symbolic and the statistical approach cast their own shadows, and each is partially blind to what lies outside its light.
These failure modes have precise analogues in the machine learning literature on out-of-distribution detection. When the data distribution shifts but the underlying generating function remains intact — covariate shift — the model’s rule R is still valid and only the residual E needs updating. This is the patchable case. When the generating function itself changes — concept drift — the rule R has broken, and no adjustment to the residual can rescue it. The failure modes described above map directly onto this distinction. The expert whose prior was wrong has committed to an R that was never correct at the relevant scale, a form of concept drift built in at design time. The network that found a local rather than global compression has an R that holds under covariate shift within its training regime but fails under concept drift when the regime moves. The connection matters because the OOD detection literature already has practical tools — calibration metrics, ensemble disagreement signals, distribution shift tests — that operationalize exactly the diagnostic this framework describes in abstract terms. The theory says what to look for. The tools exist to look for it.
— -
6. Where to inject structure
The prescriptive implication is that the central design question in any modeling system is not whether to be symbolic or statistical. It is *at what point in the compression hierarchy to inject prior structure*, and how much.
Consider the spectrum. At one extreme, you hand-specify everything: the system model, the equations, the parameter ranges, the allowable behaviors. The rule component R is almost entirely supplied by the expert, and learning is reduced to fitting a small residual. At the other extreme, you initialize a large network with no priors beyond its architecture and let it search the space of compressions from scratch. R is discovered entirely from data, and the expert contributes nothing beyond the training objective. Between these extremes lie all the actually-useful approaches: equivariant networks that bake known symmetries into the architecture, physics-informed networks that encode known conservation laws as soft constraints, foundation models that commit to the transformer’s inductive biases and pretrain on enough data for the rest to emerge, typed compositional systems in which structural invariants are enforced by the type system while dynamics are learned or simulated.
Each of these hybrid approaches is making a particular choice about *which harmonics to specify and which to discover*. The principle that organizes the spectrum is this: inject structure at the scales where your confidence in the relevant harmonics is highest, and let search handle the scales where it is not. If you know the symmetry group of the problem, tell the network. If you know the conservation laws, enforce them. If you know the compositional structure, build it into the type system. Everything you do not know with high confidence, leave to the data. The error of the pure symbolic approach is to over-specify — to inject structure at scales where the expert’s confidence was misplaced. The error of the pure statistical approach is to under-specify — to make the search do work that could have been skipped entirely.
A caveat about the confidence this prescription requires. The instruction to inject structure where confidence is highest is only as good as the confidence calibration of the person doing the injecting, and the failure modes of the previous section have already established that individual experts cannot reliably see their own blind spots from inside their own formalism. The best available corrective is to treat calibration as a community-level property rather than an individual one: inject structure where multiple independent experts, trained in different traditions and carrying different inductive biases, converge on the same relevance hierarchy. This is the human analogue of ensemble disagreement as an out-of-distribution signal — if independently trained models agree on a prediction, that agreement is evidence the prediction is reliable; if they diverge, the divergence itself is the warning. But communities trained within the same tradition can share blind spots that no amount of internal peer review will catch, just as models trained on the same data share biases that no amount of ensembling will correct. The prescriptive framework of this section is therefore conditional: it works when the mechanisms for checking confidence extend beyond any single tradition, and it degrades, in correlated and often invisible ways, when they do not.
This reframes a number of long-running debates. The question is no longer “should we use symbolic methods or neural networks,” but “at which levels of the compression hierarchy do we have reliable priors, and at which levels should we be learning from scratch.” The question is no longer “are large language models really understanding anything,” but “at what scale has their compression become valid, and how do we detect when a task requires a scale at which it has not.” The question is no longer “is MBSE obsolete in an age of learned models,” but “what is MBSE doing that learning cannot, and where does it need to hand off.”
The prescriptive move, stated most tersely: every modeling decision is a commitment to a scale and a decomposition. Choose both with your eyes open. The space of good hybrids is larger than either pure tradition has so far explored, because the pure traditions were defined by their commitment to one direction of approach rather than by any principled stance on where structure belongs.
— -
7. Where to stand
Return to the shadow.
If conservation laws are the shadow cast by symmetry, then the rules we articulate about any domain — whether as physicists, as engineers, as machine learning researchers, or as ordinary people making sense of the world — are the shadow cast by something we less often name. They are the visible trace of a compression, performed at some scale, that has found the harmonics worth keeping and the ones worth forgetting. The rule is what we see. The compression is what produced it. And the compression, in turn, depends entirely on where we chose to stand relative to the object we were trying to understand.
This is why the symbolic and the statistical are not rivals. They are two ways of standing. The expert stands close to the object, uses their accumulated intuition about which features matter, and articulates the shadow they see from that position. The network stands far away, samples the shadow from many angles at once, and infers the object from the shadows it collects. Neither is wrong, and neither has privileged access. Both are trying to triangulate the same underlying structure, and both are subject to the limits of their vantage point. What determines whether they succeed is not which tradition they come from but whether the scale of their compression matches the scale at which the result will be used.
The work of understanding any domain — and this is as true of a physicist studying a magnet as a machine learning engineer training a model as a systems engineer writing a specification — is the work of finding where to stand so that the shadow falls clearly. There is no position from which all shadows are clear at once. There is only the discipline of knowing which shadow you need, which light you are standing in, and what your own position is hiding from you.
But the shadow metaphor, like any compression, captures one thing by hiding another. It is right about what a successful compression finds — the rule, once discovered, has the character of a geometric consequence, something that was there before anyone looked for it. The Fourier basis was not invented by the network that learned modular arithmetic; it was forced by the symmetry of the group. In this sense the metaphor is faithful: what the compression reveals was already cast by the structure of the domain. Where the metaphor misleads is in the process. Shadows fall passively; compression is an active gamble. The search that finds the right harmonic could have found the wrong one, or no harmonic at all, and nothing in the shadow’s geometry tells you which case you are in. The result has the stillness of a consequence. The process that produced it had the character of a risk.
That discipline does not belong to symbolic methods or to statistical ones. It belongs to whoever is willing to ask, at each step of the modeling process, whether the compression they are committing to is the one the problem actually requires. The shadow will always be there. The question is whether you know what is casting it.
Addendum: on fixed points that are not as fixed as they look
After finishing the essay I came back to a set of notes I had made on a Sean Carroll lecture about what he calls *poetic naturalism* — the view that each coarse-grained level of description earns its own autonomous reality, justified by the renormalization group and the universality of its fixed points. A chair is a real chair, not because we have reduced it to quantum fields, but because the Renormalization Group flow of any sufficiently chair-like microphysics converges to the same macroscopic description. The fixed point exists independently of who is looking for it. Multiple microscopic theories produce the same macroscopic behavior. Irrelevant operators wash out. The high level is *ontologically independent*, not merely pragmatically useful.
Carroll’s framing is stronger than the one I used in the essay, and I think the essay is better if I say where they differ and what the stronger version implies.
The essay treats compression as a search for efficient representations. Carroll treats it as the discovery of genuinely autonomous levels of description. These are not the same claim. Under the essay’s weaker version, grokking is a clever encoding that happens to work. Under Carroll’s stronger version, grokking is the network converging on a structure that was *there* — a fixed point of the flow of all sufficient learners on that task. The Fourier basis on Z/pZ is not a parochial trick that transformers with weight decay stumbled onto; it is the relevant operator at that scale, and any reasonable complexity-penalizing search on enough data should converge to it. The striking empirical fact is that this actually seems to hold. Different architectures, different optimizers, different training regimes — when they succeed at modular arithmetic, they succeed in broadly the same way. That is a universality result, modest but real, and it does the work Carroll needs it to do.
So far, so good. The essay’s descriptive claim can be strengthened along Carrollian lines: symbolic and statistical methods converge because the fixed points are real, not because the search processes happen to share biases.
But tightening the descriptive claim this way exposes a hole in the prescriptive one, and this is the part I did not see clearly while drafting. Carroll’s ontological independence argument works cleanly in regimes where the Renormalization Group flow is smooth. Chairs do not suddenly become sensitive to quark masses. The relevance hierarchy is stable. Irrelevant operators stay irrelevant as conditions change. For most of ordinary physics this is the case, and it is why the high-level sciences get to be autonomous.
In complex systems, the hierarchy is not stable. Near a critical point, operators that were irrelevant under normal coarse-graining become relevant. The Renormalization Group flow develops singularities. Small microscopic perturbations propagate across scales rather than washing out. This is the phenomenon behind phase transitions, and it is also the phenomenon behind cascading failures in markets, regime shifts in ecosystems, viral dynamics in networks, and a good deal of what makes complex systems hard. Relevance, in these regimes, is not just scale-dependent. It is *state-dependent*. The harmonics that matter today may not be the harmonics that matter tomorrow, and the switch can be sharp.
This changes the force of section 6 in a way I want to flag. I wrote that the design discipline is to inject structure at the scales where your confidence in the relevant harmonics is highest, and let search handle the rest. That is still right as far as it goes. But the confidence itself is conditioned on the system remaining in a regime where those relevances hold. When the regime shifts — when the system approaches criticality, when a latent variable crosses a threshold, when coupling strengths change enough to alter the flow — the injected structure becomes a liability rather than an asset. The expert who hardcoded the old harmonics is now confidently wrong in a new way. The network trained under the old regime is now expensively right about the wrong thing, again. The failure modes from section 5 recur, but this time they are triggered not by mistakes in the modeling process but by the world moving underneath a model that was correctly built for a world that no longer exists.
There is a specific question that falls out of this, and it is the question I wish I had asked in the essay rather than assumed away: how do we know our cherished high-level theories are actually at fixed points, rather than in slow-moving parts of the flow? Carroll’s argument for ontological independence requires genuine fixed points. Economics, psychology, much of biology — we *treat* these as autonomous, and they mostly behave autonomously, but we have not proved they are fixed points of any well-defined flow. They might be approximate fixed points, valid in the regime we have sampled and drifting slowly in ways we have not yet noticed. The history of macroeconomic models is not comforting on this point, and the history of ecological collapse is less comforting still.
The same question applies to any model produced by any method. Is this a fixed point, or a slow-moving region of the flow that will eventually reveal itself to have been transient? The symbolic expert cannot tell from inside their own formalism. The statistical learner cannot tell from inside its training distribution. Both are in the position of someone asking whether the ground under them is bedrock or a slab of ice that happens to be moving imperceptibly.
I do not have a general answer to this question, and I am suspicious of frameworks that claim to. But the question itself suggests one addition to the discipline the main essay describes: every compression should carry an explicit claim about the timescale over which it expects to remain valid. Not a vague hedge, but a commitment — this model claims to be good for this regime, at this scale, for this long. When the horizon passes, the compression is re-examined whether or not it appears to still be working, because the whole point is that you cannot tell from inside the model whether the ground beneath it has started to move.
For most engineering work, most of the time, the fixed points are stable enough that Carroll’s autonomy argument and the essay’s compression argument both go through cleanly. You can build a bridge. You can train a classifier. You can write a systems spec and have it survive deployment.
But the cases that break are the interesting ones, and they are the cases where the framework needs to be held more loosely. The shadow that a compression casts is clear only while the light and the object stay in the same relative position. When either one moves, the shadow moves with it, and the rule that looked eternal turns out to have been the visible trace of a configuration that was only temporarily stable. This does not mean compression is not the right way to think about understanding. It means that the compression is always conditioned, always provisional, always in danger of being overtaken by a regime change that invalidates its relevances without warning.
I think this is the next question worth pulling on, and I think it is the one that connects the essay’s concerns to the harder problems of modeling living systems, social systems, and anything else that sits near enough to criticality that its fixed points are not as fixed as they look.
PS: This essay came out of my discussions and consequent edits with LLM based tools like claude code and gemini.
Citations worth reading:
Mehta, P., & Schwab, D. J. (2014). An Exact Mapping Between the Variational Renormalization Group and Deep Learning. arXiv:1410.3831.
Rissanen, J. (1978). Modeling by Shortest Data Description. Automatica, 14(5). — The introduction of MDL.
Sean Caroll’s lecture on Poetic Naturalism. This lecture really inspired me and would highly reccomend it.