Keeping up to date with LLMs, and the broader picture of AI

I was until recently working in a large multinational corporation. I was actively using some of the LLM and agentic tools, but in a very controlled environment.

That allowed me direct access and practice of state-of-the-art "consume", but I faced job focus and time constraints that did not allow me to really keep up with the "how" or "why" things are done.

Now that I have more time available, I can use my home Linux computer and learn how these things are done, under the hood.

• I don't want to spend much on exhorbitant RAM prices or the latest GPU, or middle grounds like the new Apple devices and similar offerings from NVIDIA or others to run models on local hardware.
• I do not want to spend astronomical amounts on API subscriptions either. So, for now I use my own hardware to run cloud Runpod instances that I can rent by the minute. It is too early to share insights or outcomes from things I am still learning, but watch this space…

LLMs are amazing. They feel like magic, when it comes to writing, coding, RAG, and even math apparently. Their availability has created a whole industry of "how to best use them":

• Prompt engineering (sort of came and went),
• agentic AI,
• loops,
• curating and managing context,
• all the social, ethical and business implications of using them systematically and at scale.

In spite of these amazing capabilities, and the work remaining to understand it all, LLMs are not the complete picture.

There are different types of objections or "forks in the road":

• Theoretical. The more prominent voice there is that of Yann LeCun and his joint embedding predictive architecture (JEPA) concept. This JEPA work also starts to connect with all the robotic AI work going on, by shifting learning from pixel-level reconstruction to abstract latent-space prediction, allowing robots to understand physics and plan actions efficiently.
• Practical and theoretical. There are many other "non-LLM" forms of AI, but the one that I think illustrates the topic best is the one researched by Alex Nugent — see his excellent series of blog entries and his support for memristors. There already are products available, and here's an illustration, a demo of a practical application of "other non-LLM forms of AI".

A quick pointer for anyone wanting to go deeper on JEPA specifically: Yann LeCun's original position paper lays out the architecture in more detail than a single blog post can cover, and it's worth reading alongside the talk linked above.