The past year of LLM research has focused on making the following improvements to the basic LLM machinery to overcome a number of LLM issues which are discussed elsewhere in great detail. The new LLM machinery includes:
1. Improving reasoning and increasing robustness/repeatabilty – better training sets, better optimizations.
2. Many techniques for improved training and tuning.
3. Ensuring “truth” via verifiers during runtime/testtime. Verifiers come in various “flavors” – including theorem proving in various formalisms, different kinds of domain-specific validations, tuning “Reinforcement Learning” policy generation in different ways.
Verification in Large Language Models (LLMs) ensures that generated outputs are factually accurate, logically sound, safe, and aligned with intended requirements. As models are increasingly deployed in high-stakes fields like medicine, law, and software engineering, verification has evolved from simple text comparisons into complex, multi-layered architectures. Verification strategies can be organized into four primary categories based on how and where the validation occurs:
1. Internal & Self-Verification (Prompt-Based: These techniques rely on the LLM itself to double-check its own logic and claims during or immediately after the generation process.
- Chain-of-Verification (CoVe): The LLM generates a draft response, drafts independent verification questions to cross-examine its own claims, answers those questions, and synthesizes a corrected final output.
- Verification-First (VF): The model is prompted to deliberately scrutinize a candidate or sample answer before outputting its actual solution. This reverse-reasoning reduces immediate logical lapses.
- Self-Consistency (Majority Voting): The LLM generates multiple alternate answers to the same prompt. The system then selects the most frequently occurring answer via statistical majority.
2. External Reward Models & Test-Time Scaling
Instead of using the generator, these methods rely on dedicated auxiliary layers (Reward Models) to score and filter candidate responses.
- Outcome Reward Models (ORMs): Evaluates and scores only the final answer of a candidate response. This is ideal for straightforward classification or multiple-choice results.
- Process Reward Models (PRMs): Scores every individual step within a model’s chain-of-thought. This is widely used in complex mathematical reasoning and coding tasks.
- LLM-as-a-Verifier (Continuous Logits): Rather than providing a binary “Correct/Incorrect” grade, advanced verifiers assess token-level probabilities (logits) to produce highly precise, continuous scores for grading complex agent actions.
3. Neuro-Symbolic & Formal Verification:
These methods offer strict mathematical or rule-based verification, resolving the fundamental flaw that LLMs can sometimes hallucinate during the verification process itself.
- Formal Code Verification: The LLM writes code alongside formal mathematical specifications or contracts (e.g., using tools like VeriFast or SMT solvers). External tools then mathematically prove whether the code is 100% correct.
- Rule-Based Runtime Verification (RvLLM): Intercepts outputs in real-time and evaluates them against hardcoded domain specifications (like algebraic principles or regulatory laws) to act as a definitive guardrail.
- Deterministic Symbolic Checks: Extracts structured data claims out of a model’s prose and cross-checks them against deterministic databases using symbolic logic rather than loose semantic phrasing.
4. Semantic & Similarity Verification|:
These lightweight verification methods assess textual integrity, especially when comparing outputs to golden reference standards.
- Whole-Answer Semantic Embeddings: Converts the generated text and a target source into numerical vectors to measure cosine similarity. This flags when an answer deviates in core meaning, even if it uses different vocabulary.
- Lexical & String Matching: Legacy metrics like BLEU or ROUGE track exact word overlaps. They are fast but highly rigid, often penalizing valid but uniquely phrased responses.
Comparative Summary
| Verification Type | Best For | Pros | Cons |
|---|---|---|---|
| Self-Verification (CoVe/VF) | Chatbots, Creative writing | Zero setup required | Verifier can hallucinate |
| Reward Models (PRM/ORM) | Math, Advanced logic | Highly accurate grading | Requires training/extra compute |
| Neuro-Symbolic / Formal | Software engineering, Medical | Hard math guarantees | Difficult setup, complex logic |
| Semantic Similarity | Data extraction, Legal | Fast, low computational cost | Misses deep logical errors |
In the above discussion, we can note that the only “approach” making guarantees is the neuro-symbolic formal approach. All other approaches can never provide a guarantee, do not commit to any kind of accuracy/precision and aim to compensate in terms of minimizing performance time and/or resource consumption. It requires multiple experiments to fine tune the verification process and if the knowledge evolves, the whole process has to be repeated again.
As “domains” that aim to utliize LLMs in real-world production systems increase in “complexity”, the requirements for “verification” go up exponentially – especially in mission-critical/high-stakes systems. Such domains include – finance/wealth management/investing, legal, health care etc where the cost implications of an error are asymmetric. An erroneous decision at some stage of the decision process is amplified downstream leading to much rework or in many mission critical situations to financial/property or human losses.
As the “demand” for verification goes up, adoption of LLM-centric systems even with a human-in-the-loop is a costly affair – the human needs to develop a “mental model” of how to utilize the knowledge embedded in an LLM, assess the LLM/LRMs veracity and validity in different domain contexts. Furthermore, building systems to perform this “validation/verification” seems to suggest that we need to go back to the days of building “symbolic AI systems” – of different genres – such as expert systems, rule-based systems, knowledge graphs – wherein the “domain knowledge” needs to be captured explicitly – as an ontology, rules, data schemas and more. Even the recent push towards using various types of “theorem provers” – solving pieces of domain problems – has its origins in automated program synthesis – a sub-field of symbolic AI or what is called “GOFAI”. Though the current generation of AI researchers do not want to acknowledge this explicitly for whatever reasons – we are slowly backing into old school AI – “hand-coding and constructing knowledge” – that verifies whatever the LLM generates. Using traditional AI planning frameworks for evaluating agentic workflows in “real-time” will soon be a necessity. Traditional AI planners play the role of “domain-specific” process verifiers. The interplay between LLMs and traditional AI planning is a stream of research worth further investigation. Additionally, it is important to acknowledge that Google’s acquisition of “Metabase” – a human hand-coded “fact” database – formed the scaffolding on which Google Search deployed its “summaries side bar” based on combining statistical NLP and the facts database. Though the good work done on Cyc and other symbolic systems has been discarded in main-stream Ai, I believe all of that work is essential to provide a solid spine to LLM-like systems. There is a hard line between right and wrong – It cannot be a maybe (though that may be positioned as human-like etc.) In pure economic terms, I believe we have spent billions of dollars building LLM-based systems that have achieved far less than the symbolic systems have in the last 6 decades and we are still far away from a reasonably reliable LLM-building block for intelligent systems.
Statistical LLM-like systems have been successful in enabling “perception” tasks – such as image, audio and video processing along with other kinds of signals from the physical world. However, these systems are also “embedded” in the symbolic world to provide real-world “grounding”. The notion of a “cat” – as a collection of pixels – is defined “symbolically”. The image training set requires a human to ground the symbol “cat” to a real world entity and also to the cluster of pixels – that looks like a cat. To clarify this, consider the figure below – which at a high level illustrates the “symbol” creation process. The top left illustrates how current LLM-centric AI systems are being built and the bottom right suggests why and how “symbolic AI” should play a role.
In the above figure – we first describe the overall elements of basic knowledge creation in both the images. The realworld consists of a variety of signals – signals are a class of “Symbols” (pre-defined by humans, not machines). Your everyday RGB pixel in a video frame is already symbolic – it relies on a symbolic model mapping the physics of the world of colours to numbers. Similarly audio, text, videos are all based on a symbolic substrate. Humans actively process these signals – intepret them, create abstractions, interpret them and finally “generate” symbols and symbol sequences to describe and reason about things in the realworld. The number of signals and signal types one can observe and register actively when studying and interacting with real-world phenomena is quite large. Humans filter a lot of unnecessary noise, capture only the relevant nuances and generate a new set of symbols – referring to an entity, its behaviors etc. These symbolic spaces capture causality, repeated validated true knowledge and more. These symbol spaces can then be actively “manipulated” – via logic and other derived symbol systems to make additional inferences that can then be experimentally checked with the realworld. The above describes the flow of “knowledge” in the top path. Next, these signal to symbol mappings are created – as training sets and an LLM AI system is trained to generate outputs – which is then being checked/validated verified. This is shown in the top left figure. The generative process can output a large number of alternatives many of which may not be valid in the realworld but valid in a symbol space of the world. These need to be filtered out.
In the bottom right figure, we suggest the utilization of “symbolic reasoning” – in the AI system – even before the validation step. You enforce that the AI system only generates “valid” alternatives at whatever level of abstraction. This is the neuro-symbolic system thesis – much needs to be done here. Symbolic knowledge is currently being engineered by FDEs, ontologists and more – creating a babel of non-standard mappings which is a debt we will need to cleanup in the near future. Which type of symbolic knowledge and reasoning to be used where is an open research question. LLMs for software development (including Claude and others) embed symbolic pattern matchers, regex engines and low-level compilers – all symbolic toolkits.
Given the above points, what does symbolic AI offer –
- Symbolic knowledge “reduces” the search space – both in the connectionist world and also in the “symbolic world” – as discussed above. It provides support for dealing with multiple layers of abstractions, each capturing something useful in the realworld. Secondly, it reduces the costs of search/tokenmaxxing and more. Even learning/inferring the following – “a tiger is a type of cat or a cat is a type of tiger” is extremely difficult for a connectionist system – even if cats are labelled as tigers and vice versa. It can infer – a small striped tabby cat is a tiger. This is within the realms of possibility.
- Language is a higher-order cognitive function than pure perception processes as illustrated in the evolutionary capabilities of other animals.
- Symbolic knowledge and its use is essential in the control/understanding layer of any reliable AI system. Humans can interact with an AI system only via symbols – we do not process pixels nor bits nor sound bites – as-is – but only as “language”
The anti-Knowledge Engineering efforts of the late 80s/90s in the field of AI against expert systems and other kinds of symbol systems seems to have been unwarranted. LLM/Statistical AI systems are equally brittle – if not more – given their lack of nuance and their “limited” “internal concept” base. The notion that building symbolic systems is extremely difficult and slow and costly to “capture” knowledge is *not* true as shown by the need to build simple “symbolic” systems to even “verify” what is true. Current LLM systems integrate with “procedural knowledge” via tool calling – without establishing the pre- and post- conditions of applying a tool. Tool calling works well in a digital world but can fail in a the physical world under dynamic conditions. Building higher-order symbolic abstractions and reasoning with them seems to be innate human capability rather than an LLM stack capability – though anthropomorphizing every LLM behavior seems to suggest otherwise. Even modern LLM systems such as path planners in robotics – use notions of symbolic abstractions – such as “semantic” waypoints. Systems such as Cyc and others in the same genre did achieve reasonable performance at a much higher level of abstraction and did illustrate the benefits of such symbolic systems – establishing truth, checking alignment, easy explainability and interpretabiluty, easy debugging etc. Given the above observations, it is unavoidable but acknowledge that “symbolic systems” need to be integrated with LLM-like systems to build real intelligence without human in the loop. There is no free lunch – the assumption that “knowledge can quickly be learnt from “text/video/audio” is not true. Operationalizing such knowledge requires a different kind of know-how – which is what is being engineered by building agentic systems and agent harnesses. Furthermore, generative AI systems create multiple alternatives at any abstraction layer – picking a relevant and valid one from the many across multiple abstraction layers – requires symbolic knowledge – or the end-result looks inconsistent with the real-world aka hallucinations and other phenomena. This is a major effect of all LLM-like systems – inherent in their behavior and not tunable – even zero temperature does not guarantee stable/repeatable behavior – as consistent training data is required at all levels of abstraction to solve a problem.
To complete the discussion, building software systems aims to the take the symbol systems capturing human knowledge and mechanize it in a computer – by building compilers and more. Recent advances in MLIR compiler technologies suggest a promising approach to building neuro-symbolic systems that map across multiple abstractions.
Essential take-aways from this essay are the following:
- LLM systems/connectionist systems are already “embedded” in a symbol space – although the symbol abstraction may be at a lower level. We are yet to discover mappings of these symbols to the actual physical world. So the artificial boundary between connectionist systems and traditional AI is not required and modern AI systems may benefit from a rational approach to blending the benefits of the two approaches.
- There are a wide range of problems – statistical LLMs cannot solve in a consistent/reliable/repeatable manner. Acknowledging this earlier in the AI journey makes engineering sense. Kludging inefficient solutions (Rude-Goldberg like) that solve point problems or function as POCs – that consume a lot of resources. – human, capital, energy – does not seem very judicious.
- Such a blended unified approach may yield better results rather than the run away cost/energy consumption of a “pure” statistical/training approach. Current resource hungry approaches do *not* offer a reliable production ready solution for many problems and a “core theory” of engineering AI systems is still being worked out. Imagine if in traditional engineering – say building bridges or some mechanical equipment – our approach was let us try and see – No analysis, No FEA etc. One can imagine the implications of such an approach.
Finally, we are in the very early stages of building AI systems. We do not much about how our brains/minds and more function under different conditions. For the time being – humans are the best “symbol/name” generators and we need to utilize the human defined symbol space effectively. It is early days in figuring out how “neuro-symbolic” architectures should be laid out. Much remains to be done in this AI journey.