Intro
AI agents powered by large language models have enormous potential to benefit workflows in film, tv, visual effects and animation, especially digital asset management pipelines and deployed infrastructure workflows. Implemented incorrectly, they also have the potential to substantially increase a studio’s technical debt and massively spike costs. This series of articles aims to help industry professionals, such as Technical directors (TDs) and software engineers (SWEs), learn to use agentic workflows while avoiding common pitfalls.
There’s a lot of interest in this space, but it’s rapidly evolving and very hard to keep up. There’s a lot of acronyms, new terms, and new proposals that make it challenging even if you’re working in an AI related position; I can only imagine how hard it would be if I was still working on films at the same time. In this article we’ll cover first principles, my goal being that even if you read it years after I publish the information within will still be valid despite technical advancements in the meantime.
To start, we’ll do a quick overview of the basics.
Basic Concepts
LLMs
Large Language Models, or LLMs, are the core of what most people are talking about when they discuss AI agents. LLMs are trained on massive quantities of text.
An LLM, at it’s core, is an autoregressive sequence predictor. It doesn’t generate whole sentences at once; it predicts one token at a time, appends it to its own context, and repeats the process.
Under the hood, most LLMs are using a transformer architecture. While LLMs have proven fantastic at deduction in certain areas of mathematics, they have much more trouble generating novel hypotheses from incomplete data, like Einstein inferring general relativity. For more insight into this, I suggest reading the position paper by DeepMind research Tom Zahavy, LLM’s Can’t Jump
Tokens, Encoding, and Logits
If you’ve looked at LLM pricing, you’ve probably encountered services that provide inference priced by the token. A token is typically four characters long, and acts as a basic unit of text for an LLM. Models break up your text into these tokens, encode them, run them through the neural network weights, and then decode the result. Tokens are not always necessarily whole words.
Encoding is the process of converting the tokens into a vector space that allows LLMs to train & run efficiently. Each token gets a unique id from a vocab list. The embedding model is trained to keep similar concepts “near” each other in this embedding space. Models using a transformer architecture will look at all words in the input at once, so there is also a positional encoding element added into the word vector to allow the model to understand the structure of the text.
During inference, the output of the resulting vectors get mapped back into words. This is typically done with an “unembedding matrix”, where the vector is multiplied against the entire vocabulary size to determine the probability scores of the next token. This raw score output is called a logit, and is mathematically transformed back to normal text.
To summarize: LLMs take input text, break it up, map it to unique ids, project those into multidimensional vectors, run them through the neural network weights, create a probability distribution of the most likely next token, and then convert that into english words. This is why you’ll often see services priced by input/output tokens, and APIs will make mention of something called logits.
Context
When we discuss “context” in LLMs, we’re talking about the input text the LLM can access during reasoning. Context can range from static data manually entered in input prompts, to dynamically retrieved data obtained via a tool call. Context management is critical for using LLMs on tasks; too little, and the LLM appears “dumb” as it lacks the information the user has, but if the context gets too long, LLM results degrade in quality.
Context is fundamentally about memory and state management, and typically includes the previous output generated by the LLM. In addition, the entire history of user inputs, past reasoning, tool calls, and more ends up chewing up space. As these tokens accumulate, we run into something frequently called context rot, where the underlying attention mechanism of the LLM gets diluted across a crowded vector space. This degrades performance, particularly around reasoning and matching user instruction, and increases costs as the number of input tokens increases with each output included in the context.
Scoping context appropriately is a central challenge in building systems with LLMs
Agents
When we say “agent” in the context of LLMs, we’re talking about a process where a system is using an LLM to plan, reason, and execute tools in response to a given input. Typically, this involves a loop of some sort, where the agent may react to its own output. A key distinction of an LLM agent is that the LLM is making decisions on what tools, if any, to run in response to a given input.
Agents can get pretty complex pretty quickly, but they can also be quite simple. For example, if you write a Python script that takes an input, passes the input to an LLM, exposes a basic arithmetic tool, and then prints the output, that could be considered an agent. It would be an evolution over a standard Python script, as it would allow the user to enter any text in natural language, and would likely only call the tool if the user asks it to do math in its input.
An agent loop is when we feed an agent back the result of its output and then act in response. Imagine an agent that can access a simple Python terminal. If the LLM is making a tool call to write Python, you’ll get a much better result if it can see the result of running the code, and modify it in the case of errors.
When an agent is reading the output of a tool call, it is in fact adding to its own context.
Harnesses
An agentic harness is a system that manages one or more agents, their contexts, tool calls, output, and errors. The best known class of harnesses currently are coding harnesses, such as Claude Code, Codex, and OpenCode. These harnesses manage wrangling agents for the purpose of software development, and have evolved into their own ecosystem.
A harness does not have to fundamentally be tied to a specific model, though harnesses from big AI companies like Anthropic will typically encourage (or require) use of their own proprietary models.
A harness also does not need to be exclusively for coding; Claude Design (and open-source alternative Open Design) are notable graphic design harnesses.
Harness behavior can make a big impact on performance. Specialty harnesses perform better than naive ones on benchmarks showing the impact of context, execution, and tool management on the resulting work. As a result, it’s worth researching the available options for your problem domain.
Most power users of AI likely will want to use a harness when interacting with LLMs. I recommend you start with a coding harness such as OpenCode or Pi Dev, two open-source harnesses that make it easy to use a large number of providers, including local providers.
A Note on Model Capabilities
LLMs are evolving at a breakneck pace, and it seems like every day there’s a new model release that scores amazing on one benchmark or another. Different models have differing performance on tasks; for most tasks, the biggest, newest models tend to perform best. If you last tried coding leveraging LLMs back in early 2024, for example, you likely weren’t particularly impressed. Modern state of the art models are a substantial upgrade over those earlier models, and have even been used to make breakthroughs in mathematics.
Some models are trained to tackle certain tasks better, at the expense of other capabilities. The number of places where a smaller model trained this way can beat a much larger general purpose model are decreasing, but there are still some strange jagged edges. You can think of this as shaping the latent space of the model, allowing it to spend more effort/have more knowledge of one domain at the expense of another.
I’ll talk more about model selection, as well as budget management, in a later entry in this series.
Conclusion & What’s Next
Now we have a common ground covering the basic idea of core mechanics: tokens, context boundaries, agent loops, and harnesses. AI agents aren’t magic, but they are made up of quite a bit of complicated concepts wrapped up in simpler execution code. When built thoughtfully with robust harnesses and tight context scoping, they can automate complex, non-deterministic pipeline tasks that traditional scripts struggle with. When thrown together haphazardly, they become unpredictable, costly black boxes that generate subtle bugs across your production pipeline.
In the next article in this series, we will move past the definitions and tackle common concepts in the agent programming space: how to think about context, prompts, tools, and multi-agent systems.