How Deep Is Too Deep?¶

When "Master ML" Is the Wrong Next Step¶

Jose Alekhinne / 2026-02-12

I had that voice for months. Then I spent a week debugging an agent failure that had nothing to do with ML theory and everything to do with knowing which abstraction was leaking.

This post is about when depth compounds and when it does not.

The Hierarchy Nobody Questions¶

There is an implicit stack most people carry around when thinking about AI:

At some point down that stack, you hit a comfortable plateau: the layer where you can hold a conversation but not debug a failure.

The instinctive response is to go deeper.

But that instinct hides a more important question.

Does depth still compound when the abstractions above you are moving hyper-exponentially?

The Uncomfortable Observation¶

If you squint hard enough, a large chunk of modern ML intuition collapses into older fields:

Nothing here is uniquely "AI".

Most of this math predates the term deep learning.

In some cases, by decades.

So what changed?

Same Tools, Different Regime¶

The mistake is assuming this is a new theory problem.

It is not.

It is a new operating regime.

Classical numerical methods were developed under assumptions like:

• Manageable dimensionality
• Reasonably well-conditioned objectives
• Losses that actually represent the goal

Modern ML violates all three: On purpose.

Today's models operate with millions to trillions of parameters, wildly underdetermined systems, and objective functions we know are wrong but optimize anyway.

At this scale, familiar concepts warp:

• What we call "local minima" are overwhelmingly saddle points in high-dimensional spaces
• Noise stops being noise and starts becoming structure
• Overfitting can coexist with generalization
• Bigger models outperform "better" ones

The math did not change.

The phase did.

This is less numerical analysis and more *statistical physics: Same equations, but behavior dominated by phase transitions and emergent structure.

Why Scaling Laws Feel Alien¶

In classical statistics, asymptotics describe what happens eventually.

In modern ML, scaling laws describe where you can operate today.

They do not say "given enough time, things converge".

They say "cross this threshold and behavior qualitatively changes".

This is why dumb architectures plus scale beat clever ones.

Why small theoretical gains disappear under data.

Why "just make it bigger", ironically, keeps working longer than it should.

That is not a triumph of ML theory.

It is a property of high-dimensional systems under loose objectives.

Where Depth Actually Pays Off¶

This reframes the original question.

You do not need depth because this is "AI".

You need depth where failure modes propagate upward.

I learned this building ctx. The agent failures I have spent the most time debugging were never about the model's architecture.

They were about:

• Misplaced trust: The model was confident. The output was wrong. Knowing when confidence and correctness diverge is not something you learn from a textbook. You learn it from watching patterns across hundreds of sessions.

• Distribution shift: The model performed well on common patterns and fell apart on edge cases specific to this project. Recognizing that shift before it compounds requires understanding why generalization has limits, not just that it does.

• Error accumulation: In a single prompt, model quirks are tolerable. In autonomous loops running overnight, they compound. A small bias in how the model interprets instructions becomes a large drift by iteration 20.

• Scale hiding errors: The model's raw capability masked problems that only surfaced under specific conditions. More parameters did not fix the issue. They just made the failure mode rarer and harder to reproduce.

This is the kind of depth that compounds.

Not deriving backprop.

Understanding when correct math produces misleading intuition.

The Connection to Context Engineering¶

This is the same pattern I keep finding at different altitudes.

In "The Attention Budget", I wrote about how dumping everything into the context window degrades the model's focus. The fix was not a better model. It was better curation: load less, load the right things, preserve signal per token.

In "Skills That Fight the Platform", I wrote about how custom instructions can conflict with the model's built-in behavior. The fix was not deeper ML knowledge. It was an understanding that the model already has judgment and that you should extend it, not override it.

In "You Can't Import Expertise", I wrote about how generic templates fail because they do not encode project-specific knowledge. A consolidation skill with eight Rust-based analysis dimensions was mostly noise for a Go project. The fix was not a better template. It was growing expertise from this project's own history.

In every case, the answer was not "go deeper into ML".

The answer was knowing which abstraction was leaking and fixing it at the right layer.

Agentic Systems Are Not an ML Problem¶

The mistake is assuming agent failures originate where the model was trained, rather than where it is deployed.

Agentic AI is a systems problem under chaotic uncertainty:

• Feedback loops between the agent and its environment
• Error accumulation across iterations
• Brittle representations that break outside training distribution
• Misplaced trust in outputs that look correct

In short-lived interactions, model quirks are tolerable. In long-running autonomous loops, they compound.

That is where shallow understanding becomes expensive.

But the understanding you need is not about optimizer internals.

It is about:

The depth that matters is diagnostic, not theoretical.

The Real Answer¶

Not turtles all the way down.

Go deep enough to:

• Diagnose failures instead of cargo-culting fixes
• Reason about uncertainty instead of trusting confidence
• Design guardrails that align with model behavior, not hope

Stop before:

• Hand-deriving gradients for the sake of it
• Obsessing over optimizer internals you will never touch
• Chasing theoretical purity divorced from the scale you actually operate at

This is not about mastering ML.

It is about knowing which abstractions you can safely trust and which ones leak.

A Practical Litmus Test¶

If a failure occurs and your instinct is to:

• Add more prompt text: abstraction leak above
• Add retries or heuristics: error accumulation
• Change the model: scale masking
• Reach for ML theory: you are probably (but not always) going too deep

The right depth is the shallowest layer where the failure becomes predictable.

The ctx Lesson¶

Every design decision in ctx is downstream of this principle.

The attention budget exists because the model's internal attention mechanism has real limits: You do not need to understand the math of softmax to build around it. But you do need to understand that more context is not always better and that attention density degrades with scale.

The skill system exists because the model's built-in behavior is already good: You do not need to understand RLHF to build effective skills. But you do need to understand that the model already has judgment and your skills should teach it things it does not know, not override how it thinks.

Defense in depth exists because soft instructions are probabilistic: You do not need to understand the transformer architecture to know that a Markdown file is not a security boundary. But you do need to understand that the model follows instructions from context, and context can be poisoned.

In each case, the useful depth was one or two layers below the abstraction I was working at.

Not at the bottom of the stack.

The boundary between useful understanding and academic exercise is where your failure modes live.

Closing Thought¶

Most modern AI systems do not fail because the math is wrong.

They fail because we apply correct math in the wrong regime, then build autonomous systems on top of it.

Understanding that boundary, not crossing it blindly, is where depth still compounds.

And that is a far more useful form of expertise than memorizing another loss function.

This post started as a note about whether I should take an ML course. The answer turned out to be "no, but understand why not". The meta continues.