I had the funny thought that this is exactly what a sentient AI would write "stop looking here, there is nothing to see, move along." :-)
I (like vannevar apparently) didn't feel Cyc was going anywhere useful, there were ideas there, but not coherent enough to form a credible basis for even a hypothesis of how a system could be constructed that would embody them.
I was pretty impressed by McCarthy's blocks world demo, later he and a student formalized some of the rules for creating 'context'[1] for AI to operate within, I continue to think that will be crucial to solving some of the mess that LLMs create.
For example, the early failures of LLMs suggesting that you could make salad crunchy by adding rocks was a classic context failure, data from the context of 'humor' and data from the context of 'recipes' intertwined. Because existing models have no context during training, there is nothing in the model that 'tunes' the output based on context. And you get rocks in your salad.
> there remains no evidence of its general intelligence
This seems like a high bar to reach.
We all know that symbolic AI didn't scale as well as LLMs trained on huge amounts of data. However, as you note, it also tried to address many things that LLMs still don't do well.
This is exactly correct, LLMs did scale with huge data, symbolic AI did not. So why? One of the things I periodically ask people working on LLMs is "what does a 'parameter' represent? The simplistic answer is 'it's a weight in a neural net node' but that doesn't much closer. Consider something like a bloom filter where a '0' bit represents the nth bit of all hashes of strings this filter has not seen. I would be interested in reading a paper that does a good job of explaining what a parameter ends up representing in an LLM model.[1]
I suspect that McCarthy was on to something with the context thing. Organic intelligence certainly fails in creative ways without context it would not be disqualifying to have AI fail in similarly spectacular ways.
[1] I made a bit of progress on this considering it to be the permeability for progress such that the higher the weight the easier it was to 'pass thorough' this particular neuron but the cyclic nature of the graph makes a purely topological explanation pretty obtuse :-).
LLMs did scale with huge data, symbolic AI did not. So why? [1]
Neural networks, not LLMs in particular, were just about the simplest thing that could scale - they scaled and everything else has been fine-tuning. Symbolic AI basically begins with existing mathematical models of reality and of human reason and indeed didn't scale.
The problem imo is: The standard way mathematical modeling works[2] is you have a triple of <data, model-of-data, math-formalism>. The math formalism characterizes what the data could be, how data diverges from reality etc. The trouble is that the math formalism really doesn't scale even if a given model scales[3]. So even if you were to start plugging numbers into some other math model and get a reality-approximation like an LLM, it would be a black box like an LLM because the meta-information would be just as opaque.
Consider the way Judea Pearl rejected confidence intervals and claimed probabilities were needed as the building blocks for approximate reasoning systems. But a look at human beings, animals or LLMs shows that things that "deal with reality" don't have and couldn't access to "real" probabilities.
I'd just offer that I believe that for a model to scale, the vast majority of it's parameters would have to be mathematically meaningless to us. And that's for the above reasons.
[1]. Really key point, imo
[2]. That innclude symbolic and probabilistic model "at the end of the day"
[3]. Contrast the simplicity of plugging data into a regression model versus the multitudes of approaches explaining regression and loss/error functions etc.
That's an interesting paper and worth reading. Not sure it has answered my question but I did learn some things from it that I had not considered.
This was the quote I resonated with :-)
"... the discoveries we highlight here only capture a small fraction of the mechanisms of the model."
It sometimes feels a bit like papers on cellular biology with DNA discussions in which descriptions of the enzymes and proteins involved are insightful but the mechanism that operates the reaction remains opaque.
>> This is exactly correct, LLMs did scale with huge data, symbolic AI did not. So why?
Like the rock salad you're mixing up two disparate contexts here. Symbolic AI like SAT solvers and planners is not trying to learn from data and there's no context in which it has to "scale with huge data".
Instead, what modern SAT solvers and planners do is even harder than "scaling with data" - which, after all, today means having imba hardware and using it well. SAT solving and planning can't do that: SAT is NP-complete and planning is PSPACE-complete so it really doesn't matter how much you "scale" your hardware, those are not problems you can solve by scaling, ever.
And yet, today both SAT and planning are solved problems. NP complete? Nowadays, that's a piece of cake. There are dedicated solvers for all the classical sub-categories of SAT and modern planners can solve planning problems that require sequences of thousands of actions. Hell, modern planners can even play Atari games from pixels alone, and do very well indeed [1].
So how did symbolic AI manage those feats? Not with bigger computers but precisely with the approach that the article above seems to think has failed to produce any results: heuristic search. In SAT solving, the dominant approach is an algorithm called "Conflict Driven Clause Learning", that is designed to exploit the special structure of SAT problems. In Planning and Scheduling, heuristic search was always used, but work really took off in the '90s when people realised that they could automatically estimate a heuristic cost function from the structure of a planning problem.
There are parallel and similar approaches everywhere you look at, in classical AI problems, like verification, theorem proving, etc, and that work has even produced a few Turing awards [2]. But do you hear about that work at all, when you hear about AI research? No, because it works, and so it's not AI.
But it works, it runs on normal hardware, it doesn't need "scale" and it doesn't need data. You're measuring the wrong thing with the wrong stick.
____________
[1] Planning with Pixels in (Almost) Real Time: https://arxiv.org/pdf/1801.03354 Competitive results with humans and RL. Bet you didn't know that.
[2] E.g. Pnueli for temporal logic in verification, or Clarke, Emerson and Sifakis, for model checking.
I think the problem with trying to hand-create symbolic rules for AI is that things like natural language, and the real world, are messy. Even with fuzzy rules you are never going to be able to accurately capture all the context dependencies and nuances, which may anyways be dynamic. Learning from real world data is the only realistic approach, although I don't think language models are the answer either - you need a system that is continually learning and correcting it's own errors.
CYC was an interesting experiment though. Even though it might have been expected to be brittle due to the inevitable knowledge gaps/etc, it seems there was something more fundamentally wrong with the approach for it not to have been more capable. An LLM could also be regarded as an expert system of sorts (learning its own rules from the training data), but some critical differences are perhaps that the LLM's rules are as much about recognizing context for when to apply a rule as what the rule itself is doing, and the rules are generative rather than declarative - directly driving behavior rather than just deductive closure.
Yeas, hand-coding rules doesn't work in the long run. But burning through the world's resources to approximate a huge dataset isn't a viable long-term solution for anything either.
SAT is NP-complete and planning is PSPACE-complete so it really doesn't matter how much you "scale" your hardware, those are not problems you can solve by scaling, ever.
It seems like you are not framing NP-completeness properly. An NP complete problem is simply worst case hard. Such a problem can have many solvable instances. With some distributions of randomly selected SAT problem, most instances can be quickly solvable. SAT solving contests often involve hand-constructed SATs translated from other domains and the entrants similarly add methods for these "special cases". So NP-completeness isn't a barrier to SAT-solvers scaling by itself.
I generally agree with your points, my point was mainly that the concept of "scaling" as meant in machine learning doesn't have an analogy in SAT solving and other classical AI tasks. Nobody's building large data centers to solve SAT problems and data is not "the new oil" in SAT solving, and in the other classical AI disciplines I mention above. In short, those are not data-driven fields.
> Symbolic AI like SAT solvers and planners is not trying to learn from data and there's no context in which it has to "scale with huge data".
Actually, they do. Conflict-Driven Clause Learning (CDCL) learns from conflicts encountered during working on the data. The space of inputs they are dealing with oftentimes is in the order of the number of atoms in Universe and that is huge.
I'm pretty sure most "industrial scale" SAT solvers involve both deduction and heuristics to decide which deductions to make and which to keep. At a certain scale, the heuristics have to be adaptive and then you have "induction".
I don't agree. The derivation of new clauses by Resolution is well understood as deductive and the choice of what clauses to keep doesn't change that.
Resolution can be used inductively, and also for abduction, but that's going into the weeds a bit- it's the subject of my PhD thesis. Let me know if you're in the mood for a proper diatribe :)
You know, this seems like yet another reason to allow HN users to direct message each other, or at least receive reply notifications. Dang, why can't we have nice things?
I think logical reasoning - so reasoning about logical problems, especially those with transitive relations like two way implication. A way round this is to get them to write prolog relations and then reason over them... with prolog. This isn't a fail - it's what things like prolog do, and not what things like nns do. If I was asked to solve these problems I would write prolog too.
I think quite a lot of planning.
I think scheduling - I tried something recently and GPT4 wrote python code which worked for very naive cases but then failed at any scale.
Basically though - trusted reasoning. Where you need a precise and correct answer LLM's aren't any good. They fail in the limit. But where you need a generally decent answer they are amazing. You just can't rely on it.
Whereas GOFAI you can, because if you couldn't the community thew it out and said it was impossible!
ML is good at fuzzy stuff, where you don't have a clear definition of a problem (what is spam? what is porn?), "I know it when I see it", or when you don't have a clear mathematical algorithm to solve the problem (think "distinguishing dogs and cats").
When you have both (think sorting arrays, adding numbers), traditional programming (and that includes Prolog and the symbolic AI stuff) is much better.
LLMs will always be much worse than traditional computer programs at adding large numbers, just as traditional computer programs will always be worse at telling whether the person in the image is wearing proper safety equipment.
For best results, you need to combine both. Use LLMs for the "fuzzy stuff", converting imprecise English or pictures into JSON, Python, Wolfram, Prolog or some other representation that a computer can understand and work with, and then use the computer to do the rest.
Let's say you're trying to calculate how much proteins there are per 100 grams of a product, you have a picture of the label, but the label only gives you proteins per serving and the serving size in imperial units. The naive way most programmers try is to ask an LLM to give them proteins per 100g, which is obviously going to fail in some cases. The correct way is to ask the LLM for whatever unit it likes, and then do the conversion on the backend.
I guess that's a fine distinction I don't make. If the problem requires the AI to write a prolog program to solve, and it is capable of writing the necessary prolog code, then I don't see the practical or philosophical difference from taking the transitive step and saying the AI solved it. If I asked you to solve an air traffic control problem and you did so by writing prolog, no one would try to claim you weren't capable of solving it.
Agentic LLMs can solve complicated reasoning and scheduling problems, by writing special-purpose solutions (which might resemble the things we call GOFAI). It's the nature of AGI--which LLMs assuredly are--that they can solve problems by inventing specialized tools, just as we do.
On mobile so I’m not sure how to export a chat log, but the following prompts worked with ChatGPT:
1: I need to schedule scientific operations for a space probe, given a lot of hard instrument and schedule constraints. Please write a program to do this. Use the best tool for the job, no matter how obscure.
2: This is a high-value NASA space mission and so we only get one shot at it. We need to make absolutely sure that the solution is correct and optimal, ideally with proofs.
3: Please code me up a full example, making up appropriate input data for the purpose of illustration
I got an implementation that at first glance looks correct using the MiniZinc constraint solver. I’m sure people could quibble, but I was not trying to lead the model in any way. The second prompt was because the first generated a simple python program, and I think it was because I didn’t specify that it was a high value project that needed mission assurance at the start. A better initial prompt would’ve gotten the desired result on the first try.
SAT solving, verification and model checking, automated theorem proving, planning and scheculing, knowledge representation and reasoning. Those are fields of AI research where LLMs have nothing to offer.
You could ask me, and I'll copy'n'paste for you Z3 solver, for example, stripping copyrights & rearranging code. Without any understanding how this thing work.
It will be impressive, if Claude was trained on scientific literature about SAT solvers and tutorials about programming language in question, without access to any real SAT solver code. But it is not the case.
Why do you need LLM-generated code when you can take original, which was consumed by LLM?
Or, I could ask another question: Could Claude give you SAT solver which will be 1% more effective than state-of-art in the area? We don't need another mediocre SAT solver.
I have asked Claude to solve scientific problems which I absolutely know are not in its training data, and it has done so successfully. I am not sure why people think it is only regurgitating training data. Don't be lazy and learn how it works--LLMs do generate generalized models, and employ them to solve previously unseen problems.
Would you actually use this program for real-world applications of theorem proving, e.g. validating an integrated circuit design before spending millions on its manufacturing?
I ask Claude to solve problems of similar complexity on a daily basis. A SAT solver specifically is maybe a once a week thing.
Use cases are anything, really. Determine resource allocation for a large project, or do Monte Carlo simulation of various financial and risk models. Looking at a problem that has a bunch of solutions with various trade-offs, pick the best strategy given various input constraints.
There are specialized tools out there that you can pay an arm and a leg for a license to do this, or you can have Claude one-off a project that gets the same result for $0.50 of AI credits. We live in an age of unprecedented intelligence abundance, and people are not used to this. I can have Claude implement something that would take a team of engineers months or years to do, and use it once then throw it away.
I say Claude specifically because in my experience none of the other models are really able to handle tasks like this.
What a fantastic article. One of the best on HN this year.
I first heard about Cyc as a child in a TV documentary about AI. The example involved a man shaving with an electric razor. Cyc concluded that while shaving the man was not human since the razor was an electrical device and electrical devices were not human. It really caught my imagination as a child and made me want to study AI. The magic diminished a bit once I learned more about how Cyc worked using prolog style relations and I ended up studying CS instead of AI but I still credit Cyc with sparking that initial interest in me.
Lenart seems like a strange man but we need obsessives and true believers like him to push through the various "winters". Who knows if knowledge graphs like Cyc will become relevant again in future as we seek to eliminate hallucinations from statistical learning.
I remember the very same example (re)-sparking my interest in AI back in the days.
My gut feeling says there is something in this approach that's needed to make GenAI work reliably. The brain has an associative feature, sure; but it's not very useful without filters sorting signal from nonsense, making sense of the content.
Have they been able to get Cyc to generate its own content in meaningful ways? I would expect such a system to eventually be able to derive a lot of details by itself, needing less and less spoon feeding.
Yes. Sorry. I was actually just googling this and realised this same anecdote is in cited the intro to the Deep Learning book by Goodfellow et al. Their write-up is hopefully clearer:
"For example, Cyc failed to understand a story
about a person named Fred shaving in the morning (Linde, 1992). Its inference
engine detected an inconsistency in the story: it knew that people do not have electrical parts, but because Fred was holding an electric razor, it believed the entity “FredWhileShaving” contained electrical parts. It therefore asked whether
Fred was still a person while he was shaving"
To be fair, the object "FredWhileShaving" is a state (of Fred), or simply an event, and as such it arguably is not a person in the first place. As persons are presumably neither states nor events.
You can still read his papers and textbooks. I thought he was an engaging speaker as well, and several of his lectures and interviews are available on youtube, as well as tributes and retrospectives.
A very interesting and worthwhile article (better than the comments here would suggest). However, I find it a bit of a pity that the author places so much emphasis on the assumption that the project has failed. The approach has not simply failed because the search for a solution has been going on for forty years. It took even more than forty years and costed at least as much before neural networks became really useful, and nobody would claim that the project failed because of that. And today's LLM are not really that intelligent yet. Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs. Perhaps then a greater benefit of this data will become apparent.
> Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs.
More likely, it will be made increasingly irrelevant as open alternatives to it are developed instead. The Wikipedia folks are working on some sort of openly developed interlingua that can be edited by humans, in order to populate Wikipedias in underrepresented languages with basic encyclopedic text. (Details very much TBD, but see https://en.wikipedia.org/wiki/Abstract_Wikipedia and https://meta.wikimedia.org/wiki/Abstract_Wikipedia ) This will probably be roughly as powerful as the system OP posits at some point in the article, that can generate text in both English and Japanese but only if fed with the right "common sense" to begin with. It's not clear exactly how useful logical inference on such statements might turn out to be, but the potential will definitely exist for something like that too, if it's found to be genuinely worthwhile in some way.
> made increasingly irrelevant as open alternatives to it are developed instead
Certainly interesting what these projects are going for, but it's unlikely an "open alternative", given that the degree of formalization and rigor achieved by Cyc's higher-order logic specification is likely not achievable by statistical learning, and a symbolic approach is barely achievable in a shorter time than Cyc.
It would be very surprising if the results from this approach were superior to simply machine-translating the entries from another language—because e.g. English already has so much content and contributor activity, and LLMs are already very good at translating. I can’t imagine you’d get more than a fraction of people’s interest in authoring entries in this abstract language.
LLMs are good at translating between languages that have significant amounts of written content on the internet. There are few languages in this category that do not already have correspondingly large Wikipedias.
There are plenty of languages with millions of speakers that are only rarely used in writing, often because some other language is enforced in education. If you try to use an LLM to translate into such a language, you'll just get garbage.
It's very easy for a hand-crafted template to beat an LLM if the LLM can't do the job at all.
Strictly speaking, Wikidata is an existing project which only provides a rather restrictive model for its assertions; they are not fully compositional, thus are quite far from being able to express general encyclopedic text, especially in a way that can be 'seamlessly' translated to natural language. It does provide a likely foundation for these further planned developments, though.
My daughter's PhD thesis was largely negative results. Even if the project had failed, we could learn from it if it wasn't so secretive. It could be much more open without being OSS!
It is at least as important to know which approaches do not work, but this gets significantly less press, which is not that attractive for scientists in the age of "publish or perish".
> Maybe Cycorp's knowledge base will be made generally accessible at some point
I would sooner hold my breath waiting for OpenAI to open up than Cycorp :)
> It took even more than forty years and costed at least as much before neural networks became really useful
The correct class of comparison to make with "neural networks" would be "symbolic AI" writ large. Symbolic AIs have been working quite well in some areas. Just not at all in terms of common sense reasoning, or anything approaching AGI.
If you want to keep "Cyc" in the comparison, then I would argue there is no comparison possible. Without exaggeration, there has never been a single project in AI as expensive as Cyc before 2020. Only with GPT-2 did the cost start to exceed the million USD mark. (Without exact figures, AlphaGo and Deep Blue probably also cost millions of dollars, but they unambiguously worked.)
It's also just not true that it took 40 years. Consider e.g. LeNet-5, which was up and running in 1998, and was used in ATMs to read real cheques. The main cost was 10 years of engineering stamina by LeCun's research group at Bell Labs. The finished version could be trained "for about 20 epoches over MNIST. It took 2 to 3 days of CPU time on a Silicon Graphics Origin 2000 server, using a single 200 MHz R10000 processor."
(1998 might technically be 40 years out from e.g. the inception of the perceptron in the 1950s, but if that is supposed to be our reference point for neural networks, then Cyc's reference point should be the inception of logical AIs in the same decade. And really, what use was Cyc in industry in 1998?)
It's an interesting thought experiment / philosophy / sci fi story premise though; if he spent all those years encoding his own thought processes and decision making into a program, would he have been able to create a convincing facsimile of himself and his "consciousness"? A turing test with a chatbot based on decades of self-reflection.
Stephenson's "Fall; or, Dodge in Hell" deals with the hypothetical digitization of human minds. It has some interesting ideas like some of the synthetic minds deciding to wilfully become a hivemind
You're completely moving the goalposts. And I don't find it interesting at all (I mean, I find the general subject interesting and have delved into it for decades but I don't find this sort of casual question based on no such research, trying to connect it to the wrong thing [Cyc] at all interesting) ... would he have been able to create a convincing facsimile? If and only if the encoding method were effective--that's a tautology. Was Lenat's methodology effective to that end? No, of course not, and that wasn't its intent.
"based on decades of self-reflection"
Daniel Dennett--sadly lost to us--explained in detail why "self-reflection" is not even remotely effective to this end ... our internal processes are almost entirely inaccessible to us.
You can run a version of CYC that was released online as opencyc https://github.com/asanchez75/opencyc . This is when a version of the system was posted on source forge and the GitHub has the dataset and the KB and inference engine. Note it has been written in an old version of Java.
Many years ago, I went on a quest to read as much about AM and EURISKO as I could fine. Unfortunately, Lenat kept all of his work very private (secret, even), and there was very little to read other than high level papers and articles. I always felt that was a shame, as it prevented anyone else from building on anything Lenat built. I think we’re poorer for it.
The article is a very good review of Symbolic AI, in general, not just Cyc.
I have spent a lot of time with OpenCyc in the past, but haven’t touched it in ten years.
I believe tif there is a productive future for symbolic AI that it will involve using LLMs to construct knowledge graphs, symbolic relatikns, etc. from unstructured data.
>> The article is a very good review of Symbolic AI, in general, not just Cyc.
As a "review of symbolic AI" I found it uninformed and superficial and felt that it rehashed the same old points about how symbolic AI "failed", which are disputed by the facts; specifically the fact that major symbolic AI fields like SAT solving, automated theorem proving and planning and scheduling are still going strong and have produced real-world results, so much so that e.g. SAT solving, Planning, program verification, and automated theorem proving aren't even considered "AI" anymore because they now actually work, and work very well indeed.
Technically, you are probably correct. I did find it a good walk ‘down memory lane.’ I have been working in the field since 1982 and the article made me nostalgic.
With LLM itself using those graphs to validate its answers, etc. And at some point it will be internalized into architecture as a graph attention layer.
I would argue that Lenat was at least directionally correct in understanding that sheer volume of data (in Cyc's case, rules and facts) was the key in eventually achieving useful intelligence. I have to confess that I once criticized the Cyc project for creating an ever-larger pile of sh*t and expecting a pony to emerge, but that's sort of what has happened with LLMs.
I suspect at some point the pendulum will again swing back the other way and symbolic approaches will have some kind of breakthrough and become trendy again. And, I bet it will likely have something to do with accelerating these systems with hardware, much like GPUs have done for neural networks, in order to crunch really large quantities of facts
The Bitter Lesson says "general methods that leverage computation are ultimately the most effective". That doesn't seem to rule out symbolic approaches. It does rule out anything which relies on having humans in the loop, because terabytes of data plus a dumb learning process works better than megabytes of data plus expert instruction.
(I know your message wasn't claiming that The Bitter Lesson was explicitly a counterpoint, I just thought it was interesting.)
Imho, this is wrong. Even independent of access to vast amounts of compute, symbolic methods seem to consistently underperform statistical/numerical ones across a wide variety of domains. I can't help but think that there's more to it than just brute force.
I've lost count how many times I've written the same words in this thread but: SAT Solving, Automated Theorem Proving, Program Verification and Model Checking, Planning and Scheduling. These are not domains where symbolic methods "consistently underperform" anything.
You guys really need to look into what's been going on in classical AI in the last 20-30 years. There are two large conferences that are mainly about symbolic AI, IJCAI and AAAI. Then there's all the individual conferences on the above sub-fields, like the International Conference on Automated Planning and Scheduling (ICAPS). Don't expect to hear about symbolic AI on social media or press releases from Alpha and Meta, but there's plenty of material online if you're interested.
Real AGI will need a way to reason about factual knowledge. An ontology is a useful framework for establishing facts without inferring them from messy human language.
a) The pile of LLM training data is vastly larger.
b) The data is actual human utterances in situ--these are ponies, not pony shit.
c) LLMs have no intelligence ... they channel the intelligence of a vast number of humans by pattern matching their utterances to a query. This has indeed proved useful because of how extremely well the statistical apparatus works, but the fact that LLMs have no cognitive states puts great limits on what this technology can achieve.
With Cyc, OTOH, it's not even clear what you can get out of it. The thing may well be useful if combined with LLMs, but it's under lock and key.
The big conclusions about symbolic AI that the author reaches based on this one system and approach are unwarranted. As he himself notes, "Even Ernest Davis and Gary Marcus, highly sympathetic to the symbolic approach to AI, found little evidence for the success of Cyc, not because Cyc had provably failed, but simply because there was too little evidence in any direction, success or failure."
>> they channel the intelligence of a vast number of humans by pattern matching their utterances to a query.
Just a little problem with that: to understand the utterances of a vast number of humans you need to channel it to something that can understand the utterances of humans in the first place. Just channeling it around from statistic to statistic doesn't do the trick.
Um, the "something" is the person reading the LLM's output. I'm afraid that you have completely missed the context and point of the discussion, which was not about LLMs understanding things--they understand nothing ("LLMs have no cognitive states"). But again, "because of how extremely well the statistical apparatus works", their outputs are useful to intelligent consumers who do have cognitive states--us.
That’s hilarious, but at least Llama was trained on libgen, an archive of most books and publications by humanity, no? Except for the ones which were not digitized I guess
So there is probably a big pile of Reddit comments, twitter messages, and libgen and arxiv PDFs I imagine
So there is some shit, but also painstakingly encoded knowledge (ie writing), and yeah it is miraculous that LLMs are right as often as they are
libgen is far from an archive of "most" books and publications,
not even close.
The most recent numbers from libgen itself are 2.4 million non-fiction books and 80 million science journal articles.
The Atlantic's database published in 2025 has 7.5 million books.[0]
The publishing industry estimates that many books are published each year.
As of 2010, Google counted over 129 million books[1].
At best an LLM like Llama will have have 20% of all books in its training set.
It's a miracle, but it's all thanks to the post-training. When you think of it, for so-called "next token predictors", LLMs talk in a way that almost no one actually talks, with perfect spelling and use of punctuation. The post-training somehow is able to get them to predict something along the lines of what a reasonably intelligent assistant with perfect grammar would say. LLMs are probably smarter than is exposed through their chat interface, since it's unlikely the post-training process is able to get them to impersonate the smartest character they'd be capable of impersonating.
I dunno I actually think say Claude AI SOUNDS smarter than it is, right now
It has a phenomenal recall. I just asked it about "SmartOS", something I knew about, vaguely, in ~2012, and it gave me a pretty darn good answer. On that particular subject, I think it probably gave a better answer than anyone I could e-mail, call, or text right now
But I still find it easy to stump it and get it to hallucinate, which makes it seem dumb
It is like a person with good manners, and a lot of memory, and which is quite good at comparisons (although you have to verify, which is usually fine)
But I would not say it is "smart" at coming up with new ideas or anything
I do think a key point is that a "text calculator" is doing a lot of work ... i.e. summarization and comparison are extremely useful things. They can accelerate thinking
https://ai-2027.com/ postulates that a good enough LLM will rewrite itself using rules and facts... sci-fi, but so is chatting with a matrix multiplication.
I doubt it. The human mind is a probabilistic computer, at every level. There’s no set definition for what a chair is. It’s fuzzy. Some things are obviously in the category, and some are at the periphery of it. (Eg is a stool a chair? Is a log next to a campfire a chair? How about a tree stump in the woods? Etc). This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
I find this very amusing. In philosophy of science some 20+ years ago I had a wonderful prof who went through 3(?) periods of thought. He laid out this argument, followed by the arguments seen below in this thread in various ways, in a systematic way where he convinced you that one way of thinking was correct, you took the midterm, then the next day he would lead with "everything you know is wrong, here's why.". It was beautiful.
He noted that this evolution of thought continued on until people generally argued that concepts/definitions that let you do meaningful things (your definition of meaningful, doesn't really matter what it is), are the way to go. The punchline at the very end, which happened to be the last thing I regurgitated on my last undergraduate exam, was him saying something along the lines of "Science, it beats hanging out in malls."
All this to say that if we read a little philosophy of science, that was done a long time ago (way before the class I took), things would make more sense.
I have definitely broken chairs upon sitting in them, which someone else could have sat in just fine. So it's unclear why something particular to me would change the chair-ness of an object.
Similarly, I've sat in some very uncomfortable chairs. In fact, I'd say the average chair is not a particularly comfortable one.
That's not one, but two particularities that aren't latent to the chair itself: me (the sitter), and time.
Do you really have a personal ontology that requires you to ask the tense and person acting on a thing to know what that thing is? I suspect you don't; most people don't, because it would imply that the chair wouldn't be a chair if nobody sat on it.
A stump isn't a chair until someone decides to sit on it, at that point it becomes chair _to_ that person. Chair is only capable of acting as "chair" object if constraints are met in regards to sitter.
1. I can intend to sit on a chair but fail, in which case it isn't a chair (and I didn't intend to sit on it?)
2. I can intend to have my dog sit on my chair, but my dog isn't a person and so my chair isn't a chair.
This is-use distinction you're making is fine; most people have an intuition that things "act" as a thing in relation to how they're used. But to take it a step forwards and claim that a thing isn't its nature until a person sublimates their intent towards it is very unintuitive!
(In my mind, the answer is a lot simpler: a stump isn't a chair, but it's in the family network of things that are sittable, just like chairs and horses. Or to borrow Wittgenstein, a stump bears a family resemblance to a chair.)
I'm the person who asked about the definition of a chair up thread.
Just to make a very obvious point: Nobody thinks of the definition for a chair as a particularly controversial idea. But clearly:
- We don't all agree on what a chair is (is a stump a chair or not?).
- Nobody in this thread has been able to give a widely accepted definition of the word "chair"
- It seems like we can't even agree on what criteria are admissible in the definition. (Eg, does it matter that I can sit on it? Does it matter that I can intend to sit on it? Does it matter that my dog can sit on it?)
If even defining what the word "chair" means is beyond us, I hold little hope that we can ever manually explain the concept to a computer. Returning to my original point above, this is why I think expert systems style approaches are a dead end. Likewise, I think any AI system that uses formal or symbolic logic in its internal definitions will always be limited in its capacity.
And yet, I suspect chatgpt will understand all of the nuance in this conversation just fine. Like everyone else, I'm surprised how "smart" transformer based neural nets have become. But if anything has a hope of achieving AGI, I'm not surprised that:
- Its something that uses a fuzzy, non-symbolic logic internally.
- The "internal language" for its own thoughts is an emergent result of the training process rather than being explicitly and manually programmed in.
- That it translates its internal language of thought into words at the end of the thinking / inference process. Because - as this "chair" example shows - our internal definition for what a chair is is seems clear to us. But it doesn't necessarily mean we can translate that internal definition into a symbolic definition (ie with words).
I'm not convinced that current transformer architectures will get us all the way to AGI / ASI. But I think that to have a hope of achieving human level AI, you'll always want to build a system which has those elements of thought. Cyc, as far as I can tell, does not. So of course, I'm not at all surprised its being dumped.
> Sure there is: a chair is anything upon which I can comfortably sit without breaking it.
«
It is often said that a disproportionate obsession with purely academic or abstract matters indicates a retreat from the problems of real life. However, most of the people engaged in such matters say that this attitude is based on three things: ignorance, stupidity, and nothing else.
Philosophers, for example, argue that they are very much concerned with the problems posed by real life.
Like, for instance, “what do we mean by real?”, and “how can we reach an empirical definition of life?”, and so on.
One definition of life, albeit not a particularly useful one, might run something like this: “Life is that property which a being will lose as a result of falling out of a cold and mysterious cave thirteen miles above ground level.”
This is not a useful definition, (A) because it could equally well refer to the subject’s glasses if he happens to be wearing them, and (B) because it fails to take into account the possibility that the subject might happen to fall onto, say, the back of an extremely large passing bird.
The first of these flaws is due to sloppy thinking, but the second is understandable, because the mere idea is quite clearly, utterly ludicrous.
»
A beanbag is a chair? Perhaps a chair should be something on which one can comfortably sit without breaking that has a back and four legs. I suppose then a horse would be a chair.
> Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
The counterposition to this is no more convincing: cognition is fuzzy, but it's not really clear at all that it's probabilistic: I don't look at a stump and ascertain its chairness with a confidence of 85%, for example. The actual meta-cognition of "can I sit on this thing" is more like "it looks sittable, and I can try to sit on it, but if it feels unstable then I shouldn't sit on it." In other words, a defeasible inference.
(There's an entire branch of symbolic logic that models fuzziness without probability: non-monotonic logic[1]. I don't think these get us to AGI either.)
Which word will I pick next in this sentence? Is it deterministic? I probably wouldn’t respond the same way if I wrote this comment in a different mood, or at a different time of day.
What I say is clearly not deterministic for you. You don’t know which word will come next. You have a probability distribution but that’s it. Banana.
I caught a plane yesterday. I knew there would be a plane (since I booked it) and I knew where it would go. Well, except it wasn’t certain. The flight could have been delayed or cancelled. I guess I knew there would be a plane with 90% certainty. I knew the plane would actually fly to my destination with a 98% certainty or something. (There could have been a malfunction midair). But the probability I made it home on time rose significantly when I saw the flight listed, on time, at the airport.
Who I sat next to was far less certain - I ended up sitting next to a 30 year old electrician with a sore neck.
My point is that there is so much reasoning we do all the time that is probabilistic in nature. We don’t even think about it. Other people in this thread are even talking about chairs breaking when you sit on them - every time you sit on a chair there’s a probability calculation you do to decide if the chair is safe, and will support your weight. This is all automatic.
Simple “fuzzy logic” isn’t enough because so many probabilities change as a result of other events. (If the plane is listed on the departures board, the prediction goes up!). All this needs to be modelled by our brains to reason in the world. And we make these calculations constantly with our subconscious. When you walk down the street, you notice who looks dangerous, who is likely to try and interact with you, and all sorts of things.
I think that expert systems - even with some fuzzy logic - are a bad approach because systems never capture all of this reasoning. It’s everywhere all the time. I’m typing on my phone. What is the chance I miss a letter? What is the chance autocorrect fixes each mistake I make? And so on, constantly and forever. Examples are everywhere.
To be clear, I agree that this is why expert systems fail. My point was only that non-monotonic logics and probability have equal explanatory power when it comes to unpredictability: the latter models with probability, and the former models with relations and defeasible defaults.
This is why I say the meta-cognitive explanation is important: I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case. Instead, when people use words like “likely” and “unlikely,” they’re typically expressing a defeasible statement (“typically, a stranger who approaches me on the street is going to ask me for money, but if they’re wearing a suit they’re typically a Jehovah’s Witness instead”).
> I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case.
Interesting. I don't think I agree.
I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets. For example, if someone (with no special knowledge) offered a 50/50 bet that your dining chair will break next time you sit on it, well, that sounds like a safe bet! Easy money! What about if the odds changed - so, if it breaks you give them $60, and if it doesn't break they give you $40? I'd still take that bet. What about 100-1 odds? 1000-1? There's some point where you start to say "no no, I don't want to take that bet." or even "I'd take that bet if we swap sides".
Somewhere in our minds, we hold an intuition around the probability of different events. But I think it takes a bit of work to turn that intuition into a specific number. We use that intuition for a lot of things - like, to calibrate how much surprise we feel when our expectation is violated. And to intuitively decide how much we should think through all the alternatives. If we place a bet on a coin flip, I'll think through what happens if the coin comes up heads or if it comes up tails. But if I walk into the kitchen, I don't think about the case that I accidentally stub my toe. My intuition assigns that a low enough probability that I don't think about it.
Talking about defeasible statements only scratches the surface of how complex our conditional probability reasoning is. In one sense, a transformer model is just that - an entire transformer based LLM is just a conditional probability reasoning system. The entire model of however many billions of parameters is all a big conditional probability reasoning machine who's only task is to figure out the probability distribution over the subsequent token in a stream. And 100bn parameter models are clearly still too small to hit the sweet spot. They keep getting smarter as we add more tokens. If you torture an LLM model a little, you can even get it to spit out exact probability predictions. Just like our human minds.
I think these kind of expert systems fail because they can't do the complex probability reasoning that transformer models do. (And if they could, it would be impossible to manually write out the - perhaps billions - of rules it would need to accurately reason about the world like chatgpt can.)
> I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets.
I think these are different things! I can definitely make myself think about probabilities, but that's a cognitive operation rather than a meta-cognitive one.
In other words: I think what you're describing as "a bit of work" around intuitions is our rationalization (i.e., quantification) of an process that's internally non-statistical, but defeasible instead. Defeasibility relationships can have priorities and staggerings, which we turn into fuzzy likelihoods when we express them.
My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression. I could make one up, but as you note with torturing the LLM, I'm doing it to "make progress," not because it's a true statement of probability.
(I think expert systems fail for a reason that's essentially not about probability reasoning, but dimensionality -- as the article mentions Cyc has at least 12 dimensions, but there's no reason to believe our thoughts have only or exactly these 12. There's also no reason to believe we can ever model the number of dimensions needed, given that adding dimensions to an encoded relation set is brutally exponential.)
>My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression.
LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
> LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
(Humans are imprecise, LLMs are imprecise, thermometers are imprecise, but don't stick me or my computer in an oven, please.)
>Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
Doesn't have to be like an LLM's to be probabilistic
These papers don't appear to say that: the first one describes the behavior as statistically optimal, which is exactly what you'd expect for a sound set of defeasible relations.
Or intuitively: my ability to determine whether a bird flies or not is definitely going to be statistically optimal, but my underlying cognitive process is not itself inherently statistical: I could be looking at a penguin and remembering that birds fly by default except when they're penguins, and only then if the penguin isn't wearing a jetpack. That's a non-statistical set of relations, but its external observation is modeled statistically.
>which is exactly what you'd expect for a sound set of defeasible relations.
This is a leap. While a complex system of rules might coincidentally produce behavior that looks statistically optimal in some scenarios, the paper (Ernst & Banks) argues that the mechanism itself operates according to statistical principles (MLE), not just that the outcome happens to look that way.
Moreover, it's highly unlikely, bordering on impossible, to reduce the situations the brain deals with even on a daily basis into a set of defeasible statements.
Example: Recognizing a "Dog"
Defeasible Attempt:
is_dog(X) :- has_four_legs(X), has_tail(X), barks(X), not is_cat(X), not is_fox(X), not is_robot_dog(X).
is_dog(X) :- has_four_legs(X), wags_tail(X), is_friendly_to_humans(X), not is_wolf(X).
How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
If you've tried something for decades and it's not working, and it doesn't even look like it's working and experiments with the brain suggest probabilistic inference and probabilistic inference machines work much better than the alternatives ever did, you have to face the music.
> How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
This is the dimensionality mentioned in the adjacent post, and it's true of a probabilistic approach as well: an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
(I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.)
>This is the dimensionality mentioned in the adjacent post
LLMs are only a few years old but symbolic ai was abandoned for NLP, Computer Vision etc long before that. Why ? Because the alternative was just that bad and more importantly, never seemed to really budge with effort. Companies didn't wake up one morning and pour hundreds of millions into LMs. In fact, NNs were the underdog for a very long time. They poured more and more money into it because it got better and better with investment.
There is zero reason to think even more dimensionality would do anything but waste even more time. At least the NN scalers can look back and see it work in the past. You don't even have that.
>an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
It doesn't take much to train LLMs to 'give up'. Open AI talk about this from time to time. It's just not very useful with a tendency to overcorrect. And humans hallucinate (and otherwise have weird failure modes) all the time. We just call them funny names like dunning kruger and optical illusions. Certainly less than current SOTA LLMs but it happens all the same.
>I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.
The point is not about Cyc. They're hardly the only attempt at non-monotonic logic. The point is that they should work much better than they do if there's anything to it. Again, forget recent LLMs. Even when we were doing 'humble' things like spelling error detection & correction, text compressors, voice transcription boosters, embeddings for information retrieval, recommenders, knowledge graph creation (ironically enough), machine translation services, etc these systems were not even in the conversation. They performed that poorly.
I think we're talking past each other. I'm not interested in defending symbolic AI at all; it's clear it's failed. All told, I would not say I'm particularly interested in any kind of AI.
I'm interested in theory of mind, and I think defeasibility with a huge number of dimensions is a stronger explanation of how humans behave and think than something resembling an LLM. This doesn't somehow mean that LLMs haven't "won" (they have); I just don't think they're winning at human-like cognition. This in turn does not mean we could build a better alternative, either.
That argument would hold a lot more weight if Cyc could fly. But as this article points out, decades of work and millions of dollars have utterly failed to get it off the ground.
right, but as others have pointed out the amount of money invested in Cyc is approx 2 orders of magnitude less than what was invested in LLMs. So maybe the method was OK, but it was insufficiently resourced.
Maybe, but I doubt it. Transformer models got all that investment because small transformers work and large transformers work better than small transformer models. We can extend the line and predict what happens when the model is scaled.
Small Cyc doesn't do anything useful. Large Cyc doesn't do anything useful. Why should we make huge Cyc? If we extend the line out, all evidence predicts that it'll still be useless.
> There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
Are you sure? In terms of theoretical foundations for AGI, AIXI is probabilistic but godel-machines are proof based and I think they'd meet criteria for deterministic / symbolic. Non-monotonic and temporal logics also exist, where chairness exists as a concept that might be revoked if 2 or more legs are missing. If you really want to get technical then by allowing logics with continuous time and changing discrete truth values, then you can probably manufacture a fuzzy logic where time isn't considered but truth/certainty values are continuous. Your ideas about logic might be too simple, it's more than just Aristotle
I don't know, it all seems like language games to me. The meaning is never in its grammar, but in its usage. The usage is arbitrary and capricious. I've not discovered how more nuanced forms of logics have ever really grappled with this.
In the 1980s, we used to talk about the silliness of the “grandmother neuron” - the idea that one neuron would capture an important thing, rather than a distributed representation.
> This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
That is indeed true. But we do have classic fuzzy logic, and it can be used to answer these questions. E.g. a "stool" maybe a "chair", but "automobile" is definitely not.
Maybe the symbolic logic approach could work if it's connected with ML? Maybe we can use a neural network to plot a path in the sea of assertions? Cyc really seems like something that can benefit the world if it's made open under some reasonable conditions.
> That is indeed true. But we do have classic fuzzy logic, and it can be used to answer these questions. E.g. a "stool" maybe a "chair", but "automobile" is definitely not.
I’m not convinced that classical fuzzy logic will ever solve this - at least not if every concept needs to be explicitly programmed in. What a “chair” is sort of subtly changes at a furniture store and at a campsite. Are you going to have someone explicitly, manually program all of those nuances in? No way! And without that subtlety, you aren’t going to end up with a system that’s as smart as chatgpt. Challenge me on this if you like, but we can play this game with just about any word you can name - more or less everything except for pure mathematics.
And by the way, modern ML approaches understand all of those nuances just fine. It’s not clear to me what value - if any - symbolic logic / expert systems provide that chatgpt isn’t perfectly capable of learning on its own already.
Words are used due to the absence of things. They fill an immediate experiential void and stand in for something else, because you want or need another person to evoke some fantasy to fill this absence and make understanding possible.
If you have a mind and it is a computer, then it is because of nurture, because the brain is nothing like a computer, and computers simulating language are nothing like brains.
(1) There is kind of a definition of a chair. But it's very long. Like, extremely long, and includes maybe even millions to billions of logical expressions, assuming your definition might need to use visual or geometric features of a given object to be classified as a chair (or not chair).
This is a kind of unification of neural networks (in particular LLMs) and symbolic thought: large enough symbolic thought can simulate NNs and vice versa. Indeed even the fact that NNs are soft and fuzzy does not matter theoretically, it's easy to show logical circuits can simulate soft and fuzzy boundaries (in fact, that's how NNs are implemented in real hardware! as binary logic circuits). But I think specific problems have varying degrees of more natural formulation as arithmetic, probabilistic, linear or fuzzy logic, on one hand, and binary, boolean-like logic on the other. Or natural formulations could involve arbitrary mixes of them.
(2) As humans, the actual definitions (although they may be said to exist in a certain way at a given time[1]) vary with time. We can, and do, invent new stuff all the time, and often extend or reuse old concepts. For example, I believe the word 'plug' in english likely well predates modern age, probably used to refer to original electrical power connectors. Nowadays there are USB plugs, which may not carry power at all, or audio plugs, etc. (maybe there are better examples). In any case the pioneer(s) usually did not envision all a name could be used for, and uses evolve.
(3) Words are used as tools to allow communication and, crucially, thought. There comes a need to put a fence (or maybe a mark) in abstract conceptual and logic space, and we associate that with a word. Really a word could be "anything we want to communicate", represent anything. In particular changes to the states of our minds, and states themselves. That's usually too general, most words are probably nouns which represent classifications of objects that exist in the world (like the mentioned chair) -- the 'mind state' definition is probably general enough to cover words like 'sadness', 'amazement', etc., and 'mind state transitions' probably can account for everything else.
We use words (and associated concepts) to dramatically reduce the complexity of the world to enable or improve planning. We can then simplify our tasks into a vastly simpler logical plan: even something simple like put shoes, open door, go outside, take train, get to work -- without segmenting the world into things and concepts (it's hard to even imagine thought without using concepts at all -- it probably happens instinctively), the number of possibilities involved in planning and acting would be overwhelming.
Now this puts into perspective the work of formalizing things, in particular concepts. If you're formalizing concepts to create a system like Cyc, and expect it to be cheap, simple, reliable, and function well into the future, by our observations that should fail. However, formalization is still possible, even if expensive, complex, and possibly ever changing.
There are still reasons you may want to formalize things, in particular to acquire a deeper understanding of those things, or when you're okay in creating definitions set in stone because they will be confined to a group being attentive and restrictive to their formal definitions (and not, as natural language, evolving organically according to convenience): that's the case with mathematics. The peano axioms still define the same natural numbers; and although names may be reused, you can usually specify them to a particular axiomatic definition that will never change. And thus we can keep building facts on those foundations forever -- while what is a 'plug' in natural language might change (and associated facts about plugs become invalid), we can define mathematical objects (like 'natural numbers') with unchanging properties, and ever-valid and potentially ever-growing valid facts to be known about them, reliably. So fixing concepts in stone more or less (at least when it comes to a particular axiomatization) is not such a foolish endeavor it may look like, quite the opposite! Science in general benefits from those solid foundations.
I think eventually even some concepts related to human emotions and specially ethics will be (with varying degrees of rigor) formalized to be better understood. Which doesn't mean human language should (or will) stop evolving and being fuzzy, it can do so independently of formal more rigid counterparts. Both aspects are useful.
[1] In the sense that, at a given time, you could (theoretically) spend an enormous effort to arrive at a giant rule system that would probably satisfy most people, and most objects referred to as chairs, at a given fixed time.
A similar failure of GOFAI was ABBYY's (a Russian company which was for a long time market leader in OCR software) monumental (multi decade long) attempt of creating advanced translation software entirely based on complex formal grammar parsing.
The story behind it is really interesting. This article was written by someone who worked at ABBYY:
The piece is in Russian but can (ironically) be read in good English by using e.g. the Google Translate feature inside Chrome. Which is of course entirely based on machine learning.
The story is essentially similar to Cyc: symbolic AI/logical AI/GOFAI can produce initially impressive results (ABBYY was way better than early forms of Google Translate), but symbolic approaches doesn't scale well. Big Data + machine learning wins out eventually. The piece above mentions a 2009 piece from Google which put forward this thesis. "The Unreasonable
Effectiveness of Data":
I mean if I were to oversimplify and over-abstract AGI into a long list of if / elses, that's how I'd go about it. It's just that there's A Lot to consider.
I don't have strong opinions about Cyc in either direction, but this obit feels pretty mean-spirited to me: it imputes moral failure on Lenat and others for working on symbolic AI for 40 years, without acknowledging that Lenat's basic belief ("priming the pump") is shared by ML too -- the main difference is that ML needed the pump to be primed with an overwhelming amount of compute, which grew faster than Cyc's ability to ingest facts and relations.
(This isn't to imply that I think symbolic AI "works"; only that more perspective on underlying beliefs is due.)
I don't think there's anything wrong with exploring a field for decades. There are many scientists who have a mix of successes and failures. But this guy spend his whole life and many years of other people's lives trying one single thing that never really worked. You could call that being a single-minded visionary, but I don't think it's unreasonable for others to think it either kooky or a giant waste.
A useful comparison to me here is all the alchemical efforts to turn lead into gold. Can modern physicists do that? Not economically, but sure. [1] If alchemists had just persisted, would they have gotten there too? No, it was a giant waste, and pretty loony to a modern eye. And I'd say both alchemists and a number of AI proponents both are so wrapped up in pursuing specific outcomes (gold, AGI) that they indulge in a lot of magical thinking.
I don't think this is correct. The capability for drawing useful logical inferences from something like a Cyc knowledge base is far more compute limited than just doing ML on any given amount of data. We're talking exponential (or worse!) vs. pure linear scaling. This is the real-world, practical reason why the Cyc folks eventually found no value at all in their most general inference engine, and ended up exclusively relying on their custom-authored, more constrained inference generators instead.
Again, I'm not saying Cyc's approach is correct. I'm saying that the underlying hope that made Lenat plow through the AI winter is the same one that made ML researchers plow through it. It's just that the ML researchers reached the end of the tunnel first (for some senses of first).
In spite of all of this, Cycorp is still in business, and have pivoted to healthcare automation, including apparently insurance denials. I wonder if the full Cyc knowledg base will ever end up being released to the public domain, or whether it will simply fade away into nonexistence as proprietary data?
> Cycorp is still in business, and have pivoted to healthcare automation, including apparently insurance denials.
There's a lesson here for AI entrepreneurs: If it's hard to make systems that give actually good, actionable advice, you can always pivot to authoritatively telling people what they want to be told instead. It's probably a bigger market anyway. Gordon Way would be proud.
Your tongue is in your cheek, I think, but go and read any discussion forum populated by "big 4" consultants and you'll see a high degree of worry that they are being replaced by AI. So yes, LLMs are indeed great at "authoritatively telling people what they want to be told", and they're already being used that way!
> Cyc grew to contain approximately 30 million assertions at a cost of $200 million and 2,000 person-years. Yet despite Lenat’s repeated predictions of imminent breakthrough, it never came.
That seems like pretty small potatoes compared to how much has been spent on LLMs these days.
Or to put it another way: if global funding for LLM development had been capped at $200m, how many of them would even exist?
Language models repeatedly delivered practical, real-world economic value at every step of the way from at least n-grams on. (Remember the original 'unreasonable effectiveness of data'?) The applications were humble and weren't like "write all my code for me and then immanentize the eschaton", but they were real things like spelling error detection & correction, text compressors, voice transcription boosters, embeddings for information retrieval, recommenders, knowledge graph creation (ironically enough), machine translation services, etc. In contrast, Yuxi goes through the handful of described Cyc use-cases from their entire history, and it's not impressive.
> Remember the original 'unreasonable effectiveness of data'?
That came out in 2009, correct? I wonder how much was spent on LLMs up to that point.
> In contrast, Yuxi goes through the handful of described Cyc use-cases from their entire history, and it's not impressive.
They're also not humble. Maintain a semantic database of terrorist cells? Create a tutoring AI? These seem closer to the things that LLMs are currently being used for, with middling success, after vastly more money has been pumped into the field.
Whereas most of the uses you describe for early LLMs are far more humble (spelling error detection & correction, text compressors), and also a lot more successful.
Which makes me think that CYC went first for the big targets, and fell on its face, rather than spending a few decades building up more modest accomplishments. In hindsight that would have obviously been a much better strategy, but honestly—it feels like that would have been an obviously better strategy in non-hindsight as well. I don't know why CYC went that way.
> That came out in 2009, correct? I wonder how much was spent on LLMs up to that point.
Quite a lot. Look back at the size of the teams working on language models at IBM, Microsoft, Google, etc, and think about all the decades of language model research going back to Shannon and quantifying the entropy of English. Or the costs to produce the datasets like the Brown Corpus which were so critical. And keep in mind that a lot of the research and work is not public for language models; stuff like NSA interest is obvious, but do you know what Bob Mercer did before he vanished into the black hole of Renaissance Capital? I recently learned from a great talk (spotted by OP, as it happens) https://gwern.net/doc/psychology/linguistics/bilingual/2013-... that it was language modeling!
I can't give you an exact number, of course, but when you consider the fully-loaded costs of researchers at somewhere like IBM/MS/G is usually at least several hundred thousand dollars a year and how many decades and how many authors there are on papers and how many man-years must've been spent on now-forgotten projects in the 80s and 90s to scale to billions of word corpuses to train the n-gram language models (sometimes requiring clusters), I would have to guess it's at least hundreds of millions cumulative.
> They're also not humble.
Funnily enough, the more grandiose use-cases of LMs actually were envisioned all the way back at the beginning! In fact, there's an incredible science fiction story you've never heard of which takes language models, quite literally, as the route to a Singularity, from 1943. You really have to read it to believe it: "Fifty Million Monkeys", Jones 1943 https://gwern.net/doc/fiction/science-fiction/1943-jones.pdf
> I don't know why CYC went that way.
If you read the whole OP, which I acknowledge is quite a time investment, I think Yuxi makes a good case for why Lenat culturally aimed for the 'boil the ocean' approach and how they refused to do more incremental small easily-benchmarked applications as distractions and encouraging deeply flawed paradigms and how they could maintain it for so long. (Which shouldn't be too much of a surprise. Look how much traction DL critics on HN get, even now.)
> Quite a lot. Look back at the size of the teams working on language models at IBM, Microsoft, Google, etc, and think about all the decades of language model research going back to Shannon and quantifying the entropy of English.
I wonder at what point the money spent on LLMs matched the $200 million that was ultimately spent on CYC.
> Funnily enough, the more grandiose use-cases of LMs actually were envisioned all the way back at the beginning!
Oh, I know—but those grandiose use cases still have yet to materialize, despite the time and money spent. But the smaller scale use cases have borne fruit.
> there's an incredible science fiction story you've never heard of which takes language models, quite literally, as the route to a Singularity, from 1943. You really have to read it to believe it: "Fifty Million Monkeys", Jones 1943
Thanks, I'll read that.
> If you read the whole OP, which I acknowledge is quite a time investment, I think Yuxi makes a good case for why Lenat culturally aimed for the 'boil the ocean' approach and how they refused to do more incremental small easily-benchmarked applications as distractions and encouraging deeply flawed paradigms and how they could maintain it for so long.
I read it for a chunk, but yeah, not the whole way.
> That seems like pretty small potatoes compared to how much has been spent on LLMs these days.
It seems to be a pretty high cost, at more than $6 per assertion. Wikidata - the closest thing we have to a "backbone for the Semantic Web" right now - contains around 1.6G bare assertions describing 115M real-world entities, and that's a purely volunteer project.
Global funding would never have been capped at $200M for LMs because they were obviously useful from the get go and only got more useful with more investment.
Forget CYC, Forget LLMs. We abandoned Symbolic-AI for Neural Networks in NLP long before the advent of the science-fiction esque transformer LLMs. That's how terrible they were.
It wasn't for a lack of trying either. NNs were the underdogs. Some of the greatest minds desparately wanted the symbolic approach to be a valid one and tried for literally decades, and while I wouldn't call it a 'failure', it just couldn't handle anything fuzzy without a rigidly defined problem space, which is kind of unfortunate seeing as that is the exact kind of intelligence that actually exists in the real world.
It's funny, because AI companies are currently spending fortunes on mathematicians, physicists, chemists, software engineers, etc. to create good training data.
Maybe this money would be better spent on creating a Lenat-style ontology, but I guess we'll never know.
We may. LLMs are capable, even arguably at times inventive, but lack the ability to test against ground truth; ontological reasoners can never exceed the implications of the ground truth they're given, but within that scope reason perfectly. These seem like obviously complementary strengths.
> The secretive nature of Cyc has multiple causes. Lenat personally did not release the source code of his PhD project or EURISKO, remained unimpressed with open source, and disliked academia as much as academia disliked him.
One thing that's not mentioned here, but something that I took away from Wolfram's obituary of Lenat (https://writings.stephenwolfram.com/2023/09/remembering-doug...) was that Lenat was very easily distracted ("Could we somehow usefully connect [Wolfram|Alpha and the Wolfram Language] to CYC? ... But when I was at SXSW the next year Doug had something else he wanted to show me. It was a math education game.").
My armchair diagnosis is untreated ADHD. He might have had had discussing the internals of CYC on his todo list since its first prototype, but the draft was never ready.
I thought the end goal is to make something that's way more than human in capabilities.
The thing is, if we produce something that is worse than humans (and right now LLMs are worse than humans with good search indexes at hand), there's not much point doing it. It's provably less expensive to bear, raise, and educate actual humans. And to educate a human, somehow you don't need to dump the whole internet and all pirated content ever created into their heads.
"Their topmost distinction was between things with souls and things without souls. And large trees were in the former category, whereas small trees were in the latter category…"
To me, that bit of Borges is a reminder that all human taxonomies are limited and provisional. But it seems to me that Cyc and its brethren are built around the notion that a universal taxonomy is important and achievable. I guess it's possible that a useful kind of cognition could happen that way, but it's patently not how people work. If I had gotten to the point where I was forced to define exactly when a tree got a soul, I hope I'd realize that I was barking up the wrong tree.
Lenat in his talks emphasised the infeasibility of building a universal ontology. From what I understand later Cyc modus operandi was a bunch of self consistent micro-theories but not a one huge theory of everything.
Yes, but I'm not sure he fully took in the lesson. Did they really build a bunch of separate domain-specific ontologies? Or, given that it's one system, did they build a universal ontology that's just a big mess?
The lesson I take from the fact that universal ontologies are untenable is that human cognition isn't driven by ontology, so the quest to make a thinking thing out of pile of symbolic logic is one that has no guarantee of succeeding. I think Cyc's whole project is roughly similar to the Frankensteinian notion that if you just put together the right parts and provide a vital spark, you'll get a living being. It might work and it might not, but either way it's not science; it's sympathetic magic with the trapping of science.
Well that's how LLMs got going: a critical mass of context until it ignited from its own gravitational pull.
And yes Cyc ontology wasn't consistent. Lenat's point was that it is impossible to have an ontology consistent. Which makes sense given how there is no consistence across human society or even every individual human.
I wouldn't say that LLMs have "ignited". I think they've gotten steadily better at autocomplete, but I don't think they've crossed any important threshold on the way to the sort of autonomous intelligence that Lenat was after.
I also don't think it was Lenat's original point to have an inconsistent ontology, as evidenced by his early projects. I agree that he eventually had to admit that. What I'm saying is that when he admitted that he should have recognized that it cut at the heart of what he was up to. Something I think borne out by the fact that he spent his whole life on something that didn't succeed on its own terms.
I always had my reservations about the whole AGI thing. And while I'm mightily impressed by Chat-GPT and friends, it's even clearer to me that AGI is not, and will never be, an emergent property of LLM, no matter how large the neural net. And that was likely true for Cyc as well.
I had a particular Cyc success story relayed to me years ago by a customer (not a Cycorp employee), the details of which I cannot divulge, but it was a pretty whopping success and the customer was quite happy with what Cyc had been able to do for them.
So while no AGI, it definitely seems like there was value to be had.
What do you think of the recent anthropic research on how LLMs reason? It is clear from their shallow analysis that LLMs have very serious reasoning weaknesses. I wonder if those weaknesses can be "addressed" if we build LLMs that can do deeper analysis and use RL to self-improve. LLMs improving LLMs would be a very impressive step towards AGI.
I think we are careless in how we use terms. We often say "intelligence" where me mean "sentience". We have studied intelligence for a long time and we have IQ tests that can measure it. The various LLMs (like Chat GPT and Gemini) are scoring pretty well on the IQ tests. So given that, I think we can conclude that they are intelligent, as we can measure it.
But while we have measurements for "intelligence" we don't for "sentience", "agency", "consciousness" or these other things. And I'd argue that there are lots of intelligent life on earth (take crows as an example) that are sentient to a degree that the LLMs are not. My guess is this is because of their "agency" - their drive for survival. The LLMs we have now are clearly smarter than crows and cats but not sentient in the way those animals are. So I think it's safe to say that "sentience" (whatever that is) is not an emergent property of neural net/training data size. If it were, it'd be evident already.
So Gemini/Chat GPT seem to be "intelligence", but in tool form. Very unexpected. Something I would not have believed possible 5 or 10 years ago, but there it is.
As to whether we could create a "sentient" AI, an AGI, I don't see any reason we shouldn't be able to. But it's clear to me that something else is needed, besides intelligence. Maybe it's agency, maybe it's something else (the experience of times passage?). We probably need to ways of measuring and evaluating these other things before we can progress further.
Great post that hits the sweet spot between a dry academic summary and a popular exposition.
Personally I hope that the current wave of AI is over hyped and misunderstood (so that e.g. https://ai-2027.com/ will be a comical footnote one day) and that symbolic reasoning will make a comeback in a new form.
I think there could be a next generation cyc. Current llms have too many mistakes and grounding it with an AI ontology could be really interesting. I wrote more about it here:
https://blog.jtoy.net/understanding-cyc-the-ai-database/
I wonder if it's possible to design a new formal language (something like lojban [1], with a grammar strictly based on formal logic) but with better UX so that it can be used by regular folks like me? Maybe combine something like Attempto [2] with a dedicated visual UI?
I would think a language like that could speed up knowledge base construction. Maybe it can also serve as a substitute for natural languages in some situations where we want our communication to be logically airtight.
> A heuristic arose which (as part of a daring but ill-advised experiment EURISKO was conducting) said that all machine-synthesized heuristics were terrible and should be eliminated. Luckily, EURISKO chose this very heuristic as one of the first to eliminate, and the problem solved itself.
A very thorough and interesting article -- probably the only thing you ever need read about Cyc. An example of where it shines is the way it carefully pinpoints Lenat contradicting himself on the question of converting English into Cyc's internal form -- Lenat claims that this could likely be done with NNs, but also that NNs are not capable of "common sense", which he himself (correctly) claims is necessary for language translation.
It's not without some flaws, though. The continual harping on how secretive Cyc was doesn't seem fair, given that (with the improbable exception of Meta) none of the currently leading NN-based LLMs have revealed their "source code" (weights) either. Also, I don't agree that Lenat's single-mindedness was a bad thing (I'm glad somebody explored this unpopular concept), nor even that he was necessarily wrong in believing that building a huge, curated fact base is a path to AGI -- the graph showing the exponential increase in the number of assertions in Cyc over time, despite roughly constant headcount, points towards the possibility that a critical threshold was nearby. In fact, the history of NNs is eerily similar: at first found to be promising, then abandoned, then -- against all prevailing wisdom at the time -- found to actually perform very well if you simply scaled them far more than you "should".
I always been mesmerized by Cyc (and EURISKO). I've born in late USSR, and my father (then "junior staff scientist" in natural language processing) has several translated books (borrowed from Institute's lab, as I understand mow) which were (again, as I understand now, many years alter) abbreviated translations of proceedings of some "West" conferences about AI. As I read everything about computers, I read these books too, and was deeply impressed by description of EURISKO (it worth noting, that fleet-simulation game was not mentioned, but MOS structure was). I was... 10? 12? years old.
I still think, that this approach much better than modern "multiple huge matrix by huge vector. I understand, that DNNs and LLMs won practically. But I still think that these blackbox pure-feedback based approaches are not so elegant as knowledge engineering... Practical. But in some sense "dirty". Like "dirty hacks" in code.
BTW, part about ABBYY is not exactly correct. I know this because my father became one of the founders and life-long CTO of PROMT, company which is (was? It is not clear after beginning of war, unfortunately) author of successful pre-LLM grammar-based translation systems (Stylus, later PROMT) and partner of ABBYY. ABBYY translation project stated much later than 1990s, because in 1990s ABBYY distributed PROMT technology, and ABBYY itself was (is) experts in OCR and dictionaries (dictionaries is not translation system, of course). Translation project in ABBYY started much later, when their ways with PROMT parted due to several business-related reasons.
> Lenat personally did not release the source code of his PhD project or EURISKO, remained unimpressed with open source, and disliked academia as much as academia disliked him. Most open information concerning Cyc had been deliberately removed circa 2015, at the moment when Cycorp pivoted to commercial applications.
Makes one wonder how much a research being open makes a difference in its real-world success in the current age. Cyc's competitors (LLMs etc.) arguably have a lot to attribute to open public participation for its successes. Perhaps things would have been difference had Lenat been more open with the project?
Probably not, tbh. The issue with Cyc is that it required huge amounts of manual effort to create the rules, while LLMs can learn their own rules from raw data.
There was no machine intelligence in Cyc, just human intelligence.
I think when we look at general intelligence we need to look at multiple methods of reasoning working together. Looking at children there is an instinct to blindly imitate those around them, trying to match behavior without fully understanding it, however the experiences from this behavior then feed into a symbolic understanding in the child’s mind, allowing them to reason about the situation later on
As someone who is mildly skeptical of the current wave of LLM hype and thinks it's hitting a wall, I'm noting a lot of parallels between Cyc and ML/DNN systems' founding stories. We even have tales of reward hacking and other weird meta-shenaniganery. One wonders if some renegade at Cycorp ever trained a CycL to English LLM to fake another GOFAI demo with "real" AI.
As for free lunches, the one free lunch the neural network people got was training data. i.e. being able to scrape the entire WWW and build on that[0]. But that's also a free lunch Cycorp wanted, too! Look at all the various attempts at half-open subsets of Cyc. They were half-hearted attempts to solicit training data from randos, sure, but Cycorp still begged for the free lunch anyway.
But I think the big indictment of AI - both old-fashioned and neural network driven - is that they're chasing this nigh-impossible dream of effectively replicating and replacing a person. I still see the same pattern, in both eras, of toy models working well, narrow problem solvers being effective, but then people trying to scale them up to ludicrous sizes by pumping them full of knowledge in an attempt to achieve generality. Even modern LLMs are only slightly more generalist in the sense that they don't immediately fall over and die when a question is phrased in a way that wasn't expected.
The "bitter lesson" of AI is that the only two things that work are search and learning. LLMs and Cyc both attempted to short-circuit this with massive data dumps - the proverbial "knowledge pump" that needed to be primed. Compression is a good learning substitute, so sure, you can "cheat the test" with big data. But, I mean... how many discoveries were made solely by someone mainlining Wikipedia articles like black tar heroin? At some point, you need a problem space to search, and your learning needs to be done online[1] based on the results of that search. Nobody has cracked that code yet.
Maybe the reason why general AI never took off is that there is no such thing as general knowledge. The biggest source of knowledge the AI community has provided to us is knowledge about how to make more AI.
[0] Whether or not this was legal is a different question.
[1] Specifically, while LLMs think analogously to humans (because that's an easily compressible representation) they do not learn analogously to them.
Open Mind Common Sense is a project now forgotten, a mid 2000s effort to build something like Cyc but in a more open fashion. From the MIT Media Lab, primarily Pushpinder Singh (RIP).
"The long slow failure of Lenat’s project is a strong indictment against the symbolic-logical approach to AI."
That does not follow at all ... this statement comes just after an enumeration of Lenat's attitudinal problems that plausibly played a significant role in its failure.
Are we sure that if we invested in the last ten years the same resources we invested in neural networks (research, hardware, money) the results wouldn't pay off?
What I want to ask is if we know for sure symbolic AI is a dead end and it doesn't worth to pursue that field.
Given the secrecy surrounding Cyc we don't know how close or not they got to something AGI-like. Whatever conclusion you draw will be from very little data.
From a practical standpoint Cycorp seems to have pivoted at least 10 years ago into providing knowledge and inference systems for clients. If customers are willing to pay them then perhaps we can conclude there is value in the technology. But again I'm not aware of any public examples to look at.
> From a practical standpoint Cycorp seems to have pivoted at least 10 years ago into providing knowledge and inference systems for clients. If customers are willing to pay them then perhaps we can conclude there is value in the technology.
This is exactly the kind of thing that suggests they were nowhere close to AGI. Unless you think Palantir has somehow developed AGI too and kept it secret from everyone else.
I have nothing to say about Cyc (apart to that the bitter lesson is bitter indeed, but was never debunked so far; and, also, I hate LLMs). But this line (from accompanying github) deserves some attention, IMO:
> Due to the lack of long-term stability of IA, I have taken the liberty to scrape those from IA and kept them here for safe keeping.
I mean, he is not wrong, and when I want to preserve something code-related, Github comes to mind as a safe place... which is crazy, if you think about it for a moment. And the fact that we are starting to use Github as a backup for IA is almost comically grotesque. I don't have a solution, but it's worth reminding that we sure have a problem.
My own thought process is that it's like Usenet or arXiv: The collective value of Github is so high that even if Microsoft loses interest, someone else will buy it from them or clone the content elsewhere. That said, I do wish there was an alternative that didn't involve a for-profit corporation.
Yes? I don't hate LLMs and I don't think I ever have. They are super useful and neat, I use them all the time from smolguys on my computer to Claude in the cloud.
It's like saying "I hate forks" or something, like why do you hate a generic tool that has many applications? I love that I have these things for free at home forever, my personal wiki and time tracker have both gotten way more useful ever since I baked an LLM in.
For me, the futility of GOFAI is obvious from simple introspection: is my own brain capable of chaining through vast networks of propositional statements? I wish! I can barely hold a 5 node decision tree in my head while playing a board game. At the same time, my brain is amazingly good at parsing visual and linguistic stimuli.
Still, you have to respect the man's dedication to his quixotic quest.
Yeah but helicopters fly. The problem with cyc is that it never really got off the ground.
Symbolic AI wasn't a failure but it never really worked for any problem space without rigid rules you could define and that's with some of the greatest minds trying for decades to make it happen.
At this point real-world intelligent symbolic AI only exists in fiction and our collective imaginations. Technically, no one has proven it can't work, but at some point you have to face the reality.
sure but the bigger models don’t make these trivial mistakes, and I’m not sure if translating the LLM english sentences into LISP and trying to check them is going to be more accurate than just training the models better
The bigger models avoid those mistakes by being, well, bigger. Offloading to a structured knowledgebase would achieve the same without the model needing to be bigger. Indeed, the model could be a lot smaller (and a lot less resource-intensive) if it only needed to worry about converting $LANGUAGE queries to Lisp queries and converting Lisp results back into $LANGUAGE results (where $LANGUAGE is the user's natural language, whatever that might be), rather than having to store some approximation of that knowledgebase within itself on top of understanding $LANGUAGE and understanding whatever ad-hoc query/result language it's unconsciously invented for itself.
Beyond just checking for mistakes, it would be interesting to see if Cyc has concepts that the LLMs don't or vice versa. Can we determine this by examining the models' internals?
One thing that LLMs seem to be missing, even after ingesting the whole web, is the super-basic knowledge that every human learns before the age of 3. Most of that stuff in not written anywhere, because it's totally obvious to anyone who can read. A knowledge base like Cyc can be useful to fill these gaps, because it was made for an AI with zero base knowledge and explicitly specifies even the most obvious concepts.
"As of 2025, after more than 80 years of research on Artificial Neural Networks, there remains no evidence of their general intelligence. Even after priming the pump with nearly every book ever written, all available scientific articles, and all publicly available text on the internet (and consuming more power and resources than entire countries), they routinely fail at basic tasks and generate nonsensical output and 'slop' that has already degraded the internet as a source of training data for future systems. The long slow failure of ANNs is a strong indictment against the statistical/neuromorphic approach to AI."
On top of it, Cyc's whole 40-year run expense of "200 million dollars, 2000 person-years" is... What LLM companies waste each month, for slightly less hallucinations and getting the right number of fingers most of the time?
LLMs are clearly useful, even if flawed; conversely, the author had to struggle to find uses of Cyc. Additionally, neural networks were already showing promise before the scale-ups; AlexNet was trained on consumer hardware and outperformed all other solutions to date.
I see your point, and I agree that the way that paragraph was initially posed was perhaps more dismissive than it should have been... but to disprove it, you need proof that Cyc was a working approach for anything, and said proof appears to be lacking.
There's plenty evidence of the general intelligence of LLMs. And a site that has climbed to the 6th most visited in 2 years is clearly not failing to be generally useful.
I do think the author was more scathing than necessary here though
I enjoyed this read and agree Lenat was a grifter, which is easy to see based on contracts and closed source. But I dislike how the article seems tilted towards a hit piece against search, heuristics, reasoning, symbolic approaches in AI, and even striving for explainable/understandable systems. It's a subtext throughout, so perhaps I'm misinterpreting it.. but the neats vs the scruffies thing is just not really productive, and there seems to be no real reason for the "either/or" mentality.
To put some of this into starker contrast.. 40 years, 200 million dollars, and broken promises is the cost burned on something besides ML? Wait isn't the current approach burning that kind of cash in a weekend, and aren't we proudly backdating deep-learning to ~1960 every time someone calls it "new"? Is a huge volume of inscrutable weights, with unknown sources, generated at huge costs, really "better" than closed-source in terms of transparency? Are we not very busy building agents and critics very much like Minky's society of mind while we shake our heads and say he was wrong?
This write-up also appears to me as if it were kind of punching down. A "hostile assessment" in an "obituary" is certainly easy in hindsight, especially if business is booming in your (currently very popular) neighborhood. If you didn't want to punch down, if you really want to go on record as saying logic/search are completely dead-ended and AGI won't ever touch the stuff.. it would probably look more like critiquing symbolica.ai, saying that nothing like scallop-lang / pyreason will ever find any use-cases, etc.
While I haven't been surprised to see Cyc become more and more clearly a failure, Doug Lenat was no grifter. To the very end, he (alongside a handful of others at the company) was the truest true-believer I've ever known. Cyc was his life's mission, and he never doubted that.
What do you think made Lenat work so secretively? I suspect his ideas could have advanced a lot quicker if he'd been plugged into the community, sharing and getting feedback. In research it's almost never a good move to pursue a dream in isolation.
I'd say that being a believer doesn't necessarily conflict with being a grifter. In fact the true believer is practically obligated to engage in tricks, because the ends justify the means. For example taking government contracts for anti terrorism because the problem isn't well defined, your results are hard to check, and the money is going to be good if temporary. Anything where you don't really expect to provide promised value is always worth it because you're going to "pay it back" with big results later. For more recent stuff along the same lines, colonizing mars and getting away from oil are like that too, because for the true believer stuff like this is worth a little stock manipulation or whatever else is required.
Grifting suggests a level of cynicism, that one knows that one is selling snake oil.
I don’t know Lenat and don’t have an opinion one way or another. But be careful suggesting someone is grifting verses just believing in an idea that ultimately doesn’t come to fruition.
> even striving for explainable/understandable systems
It's been almost 6-8000 years since the advent of writing and we still cannot explain or understand human intelligence and yet we expect to be able to understand a machine that is close to or surpasses human intelligence? Isn't the premise fundamentally flawed?
I think I'd remain interested in more conclusive proof one way or the other, since by your logic everything that's currently unknown is unknowable.
Regardless of whether the project of explainable / understandable succeeds though, everyone should agree it's a worthy goal. Unless you like the idea of stock-markets, resource planning for cities and whole societies under the control of technology that's literally indistinguishable from oracles speaking to a whispering wind. I'd prefer someone else is able to hear/understand/check their math or their arguments. Speaking of 6-8000 years since something happened, oracles and mystical crap like that should be forgotten relics of a bygone era rather than an explicit goal for the future
It is actually incredibly silly to expect full explain ability as a goal because any system sufficiently intelligent to do basic arithmetic will have behavior that is inexplicable.
I like the guys general ideas about research but um. Did you see the section describing contracts? Article states 50% of funding came from the military. People would be freaking out if they heard the same about Google, Facebook, or OpenAI.. for good reason.
I'm not a fan of weaponizing AI, and I think that's what we're talking about. Either it was a glorified CMS, in which case the presentation as AI was dishonest and cynical. Or it really was AI, in which case it was weaponized research.
If we're talking about graves, then it might be good to also consider all of the ones that you're not mentioning, the ones presumably resulting from the details about where the money came from. How many? How many of those deserved it and how many were bad inferences? I guess we'll never know.
Oh, OK. Well I don't call that a grifter, just an ordinary, garden variety, techie. Many (not all) do that; actively seek funding from militaries for their work.
E.g., just this week MS fired two people for protesting the use of Azure to power the Palestinian Genocide [1].
When people talk about the military-industrial complex, what they really should be talking about is the military-FAANG complex. AI and military intelligence are both the same sad joke.
Lenat was no different in that, so I don't think it's fair to call him a grifter. I do think it's fair to call him out on being an asshole who put money above peoples' lives.
Btw, I've released some of my free stuff under a modified GNU 3.0 with an added clause that prohibits its use for military applications. I've been told that makes it "non-free" and it seems that's a bad thing. Lenat is only one nerd in a long line of nerds that need to think very hard about the ethics of their work.
This is the best overview I have ever seen, and I have had a passing interest in Cyc since I first read about EURISKO. Hands down, this is great: thorough, historical ... perhaps a little more negative than I would like.
My personal "take," which is worth nothing, is that something like Cyc is necessary but not sufficient for a serious AGI. Many other components will probably be required.
Yes, we are gonna need natural language parsing, and a good one, unless we can make the whole world speak in lojban. It would certainly speed up the collection of assertions.
I suspect embodiment is also a requisite, wherein one can acquire experiences and test premises. The ability to explore your surroundings, to move, to investigate. To jam your sensory cluster where it does not belong. To drop a block and see it fall. To drop a balloon and see it rise -- surely that must suggest an avenue of investigation.
I would be surprised if neural networks will not make a showing. Recognition through the fuzziness and quirks of reality is needed before you can sufficiently abstract into various rules. That isn't a new species of animal, it's a cat without fur and with an unusual number of limbs (Sphynx with a missing leg). Recognize, then expand the knowledge: cats usually have fur, but some breeds do not. Cats usually have four limbs, but they can be lost. But before you can say that, you still have to recognize the catness of the beast before you.
Another probable component: emotion. e-motion. Motion. Drive. In a way, hunger is an emotion, as is a full bladder. These are prompts to do something that one might return to equilibrium and focus on the higher level problems. Buildup of carbon dioxide, that one surely prompts you. These are currently tied to our embodiment, but parallels can be worked out, such as Opportunity's last ... words ... "My battery is low and it's getting dark." Right now, our attempts at intelligences are still reactive. Waiting for a prompt, waiting for a problem to be put before them. I suspect a strange loop (or really a series of interconnected ones) might be one of the final components, with the emotions serving as a way to keep it from lapsing into catatonia. Boredom would be one. Most of our human emotions would be counter-productive, but probably not all, and there may be analogues.
As a subcomponent of that, I believe in laughter. Babies laugh a lot. We laugh at jokes. No, we laugh at new jokes; old jokes grow progressively less and less funny. Laughter seems to be often prompted by a new connection, and it is both a reward and a communication.
I have covered both textual ingestion and experiential evidence, but let's throw in another one: I expect that the ability to examine an expert system and replicate it, to incorporate it into its mind, is one of them. Think about those old flowcharts in hardware books which enable you to diagnose a computer with boot issues. We can do it and I expect an AGI ought to be able to examine other rulesets for ingestion as well.
In short, I am suggesting that our first AGI may require all of the things a human does to be intelligent. We may find that we can do without one or two components. While I most certainly do not want a Human in a Box to be the end product, for a number of reasons, it does seem reasonable to me that we ought to incorporate the different things we already know about the human mind into something we are hoping to be an equivalent.
However, I strongly suspect that giving it full access to something like Cyc would beat just chucking encyclopedias at it and telling it to read.
Yep, they completely missed the boat. They tried to use concepts without actually modeling concepts, making a huge mess of contradicting statements which actually didn't model the world. Using a word in a statement does not a concept make!
>> As of 2025, 9 years after the knowledge pump had been primed, there is still no sign that Cyc would ever achieve general intelligence. The long slow failure of Lenat’s project is a strong indictment against the symbolic-logical approach to AI.
Really? CYC, a strong indictment against SAT solving, though unrelated to it? A strong indictment against automated theorem proving, though unrelated to it? A strong indictment against planning and scheduling, though unrelated to it? A strong indictment against program verification and model checking, though unrelated to it? A strong indictment against Knowledge Representation and Reasoning, though unrelated to it?
Or are those "symbolic-logical approaches to AI"? Some of those fields use heuristic search, but then so does Neural Networks research. What's gradient optimisation? It's a search algorithm that follows the gradient of a function based on the strong assumption that the function has its optimum at the location of the searched-for datum. So, a search algorithm; with a heuristic cost function.
How about probabilistic AI? Is that "symbolic-logical AI"? I mean the Bayesian calculus is a form of logic after all, and probabilities are symbolic right up to the time when they turn into statistics when you have to give them values [1].
How about neural networks again? Everyone will tell you they are "sub-symbolic" but in truth their "continuous" weights are finite-precision floating point numbers, so they're basically symbols, with a very large vocabulary. And in any case Neural Nets, like everything else, are running on a computer, a symbol manipulation device based on Boolean algebra and logic gates. Is that symbolic, logical, or neither?
See what happens when you make up new terminology as you go along and then pretend like it's some kind of agreed-upon definition that everyone agrees upon? Yes I know. My first language is not English either, but the author has clearly gone and er delved into old papers about CYC so I don't see that they have any excuse to make all those unsubstantiated, and unsupported, assumptions about what "symbolic-logical AI" is.
And as for CYC and Lenat: it's easy to disparage a man's work when he's dead and can't defend it. It's easier still if you're just repeating the same talking points that has circulated in the grapevine for who knows how long: blah blah CYC failed, blah blah symbolic AI failed blah. The fact of the matter is that CYC is closed source, very few people can use it, and nobody, outside of Cycorp, knows its true capabilities. A bit like "OpenAI"'s latest "reasoning" models, all we can do is speculate and pretend we know what we're talking about, when we clearly don't. I mean they. The author.
_________________
[1] Which is what you have to do if you want to use probabilities in the real world. And if you feed the Mogwai after midnight they turn into Gremlins.
First, apologies for writing "symbolic-logical AI". It should have been "symbolic-logical AGI".
Second, I agree that the failure of the symbolic-logical approach to AGI is not justified in-text. I believe that it will only be a small component to the first AGI system, as an API for the AGI to use some tools like a calculator, a proof verifier, etc. A way to invoke the "crystalized intelligence" as it is. Nevertheless, since it is not justified in-text, and a justification would probably as a short essay, and is not necessary for the essay itself, it has been removed.
They are specifically talking about "general intelligence" here, something which hasn't been achieved and so we can probably classify all attempts towards it as failures at this point.
I don't think that the author meant to write off the entire field of symbolic reasoning because Cyc didn't achieve general intelligence but I think it's fair to say that the lack of progress Cyc made towards it does make it less likely that purely symbolic reasoning will lead us to AGI. That's not to say it might not play a part in AGI or that it doesn't have other uses.
I had the funny thought that this is exactly what a sentient AI would write "stop looking here, there is nothing to see, move along." :-)
I (like vannevar apparently) didn't feel Cyc was going anywhere useful, there were ideas there, but not coherent enough to form a credible basis for even a hypothesis of how a system could be constructed that would embody them.
I was pretty impressed by McCarthy's blocks world demo, later he and a student formalized some of the rules for creating 'context'[1] for AI to operate within, I continue to think that will be crucial to solving some of the mess that LLMs create.
For example, the early failures of LLMs suggesting that you could make salad crunchy by adding rocks was a classic context failure, data from the context of 'humor' and data from the context of 'recipes' intertwined. Because existing models have no context during training, there is nothing in the model that 'tunes' the output based on context. And you get rocks in your salad.
[1] https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...
> there remains no evidence of its general intelligence
This seems like a high bar to reach.
We all know that symbolic AI didn't scale as well as LLMs trained on huge amounts of data. However, as you note, it also tried to address many things that LLMs still don't do well.
This is exactly correct, LLMs did scale with huge data, symbolic AI did not. So why? One of the things I periodically ask people working on LLMs is "what does a 'parameter' represent? The simplistic answer is 'it's a weight in a neural net node' but that doesn't much closer. Consider something like a bloom filter where a '0' bit represents the nth bit of all hashes of strings this filter has not seen. I would be interested in reading a paper that does a good job of explaining what a parameter ends up representing in an LLM model.[1]
I suspect that McCarthy was on to something with the context thing. Organic intelligence certainly fails in creative ways without context it would not be disqualifying to have AI fail in similarly spectacular ways.
[1] I made a bit of progress on this considering it to be the permeability for progress such that the higher the weight the easier it was to 'pass thorough' this particular neuron but the cyclic nature of the graph makes a purely topological explanation pretty obtuse :-).
> LLMs did scale with huge data, symbolic AI did not.
Symbolic AI have not had a privilege to be applied or "trained" with huge data. 30 millions assertions is not a big number.
This is correct. Those 30M assertions were basically entered by hand.
LLMs did scale with huge data, symbolic AI did not. So why? [1]
Neural networks, not LLMs in particular, were just about the simplest thing that could scale - they scaled and everything else has been fine-tuning. Symbolic AI basically begins with existing mathematical models of reality and of human reason and indeed didn't scale.
The problem imo is: The standard way mathematical modeling works[2] is you have a triple of <data, model-of-data, math-formalism>. The math formalism characterizes what the data could be, how data diverges from reality etc. The trouble is that the math formalism really doesn't scale even if a given model scales[3]. So even if you were to start plugging numbers into some other math model and get a reality-approximation like an LLM, it would be a black box like an LLM because the meta-information would be just as opaque.
Consider the way Judea Pearl rejected confidence intervals and claimed probabilities were needed as the building blocks for approximate reasoning systems. But a look at human beings, animals or LLMs shows that things that "deal with reality" don't have and couldn't access to "real" probabilities.
I'd just offer that I believe that for a model to scale, the vast majority of it's parameters would have to be mathematically meaningless to us. And that's for the above reasons.
[1]. Really key point, imo [2]. That innclude symbolic and probabilistic model "at the end of the day" [3]. Contrast the simplicity of plugging data into a regression model versus the multitudes of approaches explaining regression and loss/error functions etc.
>I would be interested in reading a paper that does a good job of explaining what a parameter ends up representing in an LLM model.
https://distill.pub/2020/circuits/ https://transformer-circuits.pub/2025/attribution-graphs/bio...
That's an interesting paper and worth reading. Not sure it has answered my question but I did learn some things from it that I had not considered.
This was the quote I resonated with :-)
"... the discoveries we highlight here only capture a small fraction of the mechanisms of the model."
It sometimes feels a bit like papers on cellular biology with DNA discussions in which descriptions of the enzymes and proteins involved are insightful but the mechanism that operates the reaction remains opaque.
>> This is exactly correct, LLMs did scale with huge data, symbolic AI did not. So why?
Like the rock salad you're mixing up two disparate contexts here. Symbolic AI like SAT solvers and planners is not trying to learn from data and there's no context in which it has to "scale with huge data".
Instead, what modern SAT solvers and planners do is even harder than "scaling with data" - which, after all, today means having imba hardware and using it well. SAT solving and planning can't do that: SAT is NP-complete and planning is PSPACE-complete so it really doesn't matter how much you "scale" your hardware, those are not problems you can solve by scaling, ever.
And yet, today both SAT and planning are solved problems. NP complete? Nowadays, that's a piece of cake. There are dedicated solvers for all the classical sub-categories of SAT and modern planners can solve planning problems that require sequences of thousands of actions. Hell, modern planners can even play Atari games from pixels alone, and do very well indeed [1].
So how did symbolic AI manage those feats? Not with bigger computers but precisely with the approach that the article above seems to think has failed to produce any results: heuristic search. In SAT solving, the dominant approach is an algorithm called "Conflict Driven Clause Learning", that is designed to exploit the special structure of SAT problems. In Planning and Scheduling, heuristic search was always used, but work really took off in the '90s when people realised that they could automatically estimate a heuristic cost function from the structure of a planning problem.
There are parallel and similar approaches everywhere you look at, in classical AI problems, like verification, theorem proving, etc, and that work has even produced a few Turing awards [2]. But do you hear about that work at all, when you hear about AI research? No, because it works, and so it's not AI.
But it works, it runs on normal hardware, it doesn't need "scale" and it doesn't need data. You're measuring the wrong thing with the wrong stick.
____________
[1] Planning with Pixels in (Almost) Real Time: https://arxiv.org/pdf/1801.03354 Competitive results with humans and RL. Bet you didn't know that.
[2] E.g. Pnueli for temporal logic in verification, or Clarke, Emerson and Sifakis, for model checking.
I think the problem with trying to hand-create symbolic rules for AI is that things like natural language, and the real world, are messy. Even with fuzzy rules you are never going to be able to accurately capture all the context dependencies and nuances, which may anyways be dynamic. Learning from real world data is the only realistic approach, although I don't think language models are the answer either - you need a system that is continually learning and correcting it's own errors.
CYC was an interesting experiment though. Even though it might have been expected to be brittle due to the inevitable knowledge gaps/etc, it seems there was something more fundamentally wrong with the approach for it not to have been more capable. An LLM could also be regarded as an expert system of sorts (learning its own rules from the training data), but some critical differences are perhaps that the LLM's rules are as much about recognizing context for when to apply a rule as what the rule itself is doing, and the rules are generative rather than declarative - directly driving behavior rather than just deductive closure.
Yeas, hand-coding rules doesn't work in the long run. But burning through the world's resources to approximate a huge dataset isn't a viable long-term solution for anything either.
SAT is NP-complete and planning is PSPACE-complete so it really doesn't matter how much you "scale" your hardware, those are not problems you can solve by scaling, ever.
It seems like you are not framing NP-completeness properly. An NP complete problem is simply worst case hard. Such a problem can have many solvable instances. With some distributions of randomly selected SAT problem, most instances can be quickly solvable. SAT solving contests often involve hand-constructed SATs translated from other domains and the entrants similarly add methods for these "special cases". So NP-completeness isn't a barrier to SAT-solvers scaling by itself.
I generally agree with your points, my point was mainly that the concept of "scaling" as meant in machine learning doesn't have an analogy in SAT solving and other classical AI tasks. Nobody's building large data centers to solve SAT problems and data is not "the new oil" in SAT solving, and in the other classical AI disciplines I mention above. In short, those are not data-driven fields.
> Symbolic AI like SAT solvers and planners is not trying to learn from data and there's no context in which it has to "scale with huge data".
Actually, they do. Conflict-Driven Clause Learning (CDCL) learns from conflicts encountered during working on the data. The space of inputs they are dealing with oftentimes is in the order of the number of atoms in Universe and that is huge.
"Learning" in CDCL is a misnomer: the learning process is Resolution and it's deductive (reasoning) not inductive (learning).
You invented a new kind of learning that somewhat contradicts usual definition [1] [2].
"Learning" in CDCL is perfectly in line of "gaining knowledge."I'm pretty sure most "industrial scale" SAT solvers involve both deduction and heuristics to decide which deductions to make and which to keep. At a certain scale, the heuristics have to be adaptive and then you have "induction".
I don't agree. The derivation of new clauses by Resolution is well understood as deductive and the choice of what clauses to keep doesn't change that.
Resolution can be used inductively, and also for abduction, but that's going into the weeds a bit- it's the subject of my PhD thesis. Let me know if you're in the mood for a proper diatribe :)
Take a look at Satisfaction-Driven Clause Learning [1].
[1] https://www.cs.cmu.edu/~mheule/publications/prencode.pdf
I'd love a diatribe if you're still following this post.
As would I.
You know, this seems like yet another reason to allow HN users to direct message each other, or at least receive reply notifications. Dang, why can't we have nice things?
Oh, hi guys. Sorry just saw this.
Oh gosh I gotta do some work today, so no time to write what I wanted. Maybe watch this space? I'll try to make some time later today.
> No, because it works, and so it's not AI
This is an important point. Hard "AI" problems are no longer "AI" once we have good algorithms and/or heuristics to solve them.
Well, we haven't tried symbolic AI with huge amounts of data. It's a hard problem.
(And ironically this problem is much easier now that we have LLMs to help us clean and massage textual data.)
Such as what? What can GOFAI do well that LLMs still cannot?
I think logical reasoning - so reasoning about logical problems, especially those with transitive relations like two way implication. A way round this is to get them to write prolog relations and then reason over them... with prolog. This isn't a fail - it's what things like prolog do, and not what things like nns do. If I was asked to solve these problems I would write prolog too.
I think quite a lot of planning.
I think scheduling - I tried something recently and GPT4 wrote python code which worked for very naive cases but then failed at any scale.
Basically though - trusted reasoning. Where you need a precise and correct answer LLM's aren't any good. They fail in the limit. But where you need a generally decent answer they are amazing. You just can't rely on it.
Whereas GOFAI you can, because if you couldn't the community thew it out and said it was impossible!
This has always been the case with ML.
ML is good at fuzzy stuff, where you don't have a clear definition of a problem (what is spam? what is porn?), "I know it when I see it", or when you don't have a clear mathematical algorithm to solve the problem (think "distinguishing dogs and cats").
When you have both (think sorting arrays, adding numbers), traditional programming (and that includes Prolog and the symbolic AI stuff) is much better.
LLMs will always be much worse than traditional computer programs at adding large numbers, just as traditional computer programs will always be worse at telling whether the person in the image is wearing proper safety equipment.
For best results, you need to combine both. Use LLMs for the "fuzzy stuff", converting imprecise English or pictures into JSON, Python, Wolfram, Prolog or some other representation that a computer can understand and work with, and then use the computer to do the rest.
Let's say you're trying to calculate how much proteins there are per 100 grams of a product, you have a picture of the label, but the label only gives you proteins per serving and the serving size in imperial units. The naive way most programmers try is to ask an LLM to give them proteins per 100g, which is obviously going to fail in some cases. The correct way is to ask the LLM for whatever unit it likes, and then do the conversion on the backend.
I guess that's a fine distinction I don't make. If the problem requires the AI to write a prolog program to solve, and it is capable of writing the necessary prolog code, then I don't see the practical or philosophical difference from taking the transitive step and saying the AI solved it. If I asked you to solve an air traffic control problem and you did so by writing prolog, no one would try to claim you weren't capable of solving it.
Agentic LLMs can solve complicated reasoning and scheduling problems, by writing special-purpose solutions (which might resemble the things we call GOFAI). It's the nature of AGI--which LLMs assuredly are--that they can solve problems by inventing specialized tools, just as we do.
Can you show us a log from when you gave an LLM a scheduling problem or something and it decided to solve it with Prolog or Z3 or something?
On mobile so I’m not sure how to export a chat log, but the following prompts worked with ChatGPT:
1: I need to schedule scientific operations for a space probe, given a lot of hard instrument and schedule constraints. Please write a program to do this. Use the best tool for the job, no matter how obscure.
2: This is a high-value NASA space mission and so we only get one shot at it. We need to make absolutely sure that the solution is correct and optimal, ideally with proofs.
3: Please code me up a full example, making up appropriate input data for the purpose of illustration
I got an implementation that at first glance looks correct using the MiniZinc constraint solver. I’m sure people could quibble, but I was not trying to lead the model in any way. The second prompt was because the first generated a simple python program, and I think it was because I didn’t specify that it was a high value project that needed mission assurance at the start. A better initial prompt would’ve gotten the desired result on the first try.
"Tried to address" is not the same as "can do well."
I was responding to PP, but some other (maybe obvious?) examples are logical reasoning and explainability.
As PP suggests, some of the classical symbolic ideas may be applicable or complementary to current approaches.
See also (Lenat's final paper):
D. Lenat, G. Marcus, "Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc", https://arxiv.org/abs/2308.04445
SAT solving, verification and model checking, automated theorem proving, planning and scheculing, knowledge representation and reasoning. Those are fields of AI research where LLMs have nothing to offer.
I can ask Claude 3.7 to write me a program that does SAT solving, theorem proving, or scheduling, and it generally gets it right on the first try.
You could ask me, and I'll copy'n'paste for you Z3 solver, for example, stripping copyrights & rearranging code. Without any understanding how this thing work.
It will be impressive, if Claude was trained on scientific literature about SAT solvers and tutorials about programming language in question, without access to any real SAT solver code. But it is not the case.
Why do you need LLM-generated code when you can take original, which was consumed by LLM?
Or, I could ask another question: Could Claude give you SAT solver which will be 1% more effective than state-of-art in the area? We don't need another mediocre SAT solver.
I have asked Claude to solve scientific problems which I absolutely know are not in its training data, and it has done so successfully. I am not sure why people think it is only regurgitating training data. Don't be lazy and learn how it works--LLMs do generate generalized models, and employ them to solve previously unseen problems.
Would you actually use this program for real-world applications of theorem proving, e.g. validating an integrated circuit design before spending millions on its manufacturing?
I do, yes.
Demonstrate.
It would take you all of 5 seconds to try in Claude yourself. I do this work on a daily basis; I know its value.
Do you mean you create SAT solvers with Claude on a daily basis? What is the use case for that?
I ask Claude to solve problems of similar complexity on a daily basis. A SAT solver specifically is maybe a once a week thing.
Use cases are anything, really. Determine resource allocation for a large project, or do Monte Carlo simulation of various financial and risk models. Looking at a problem that has a bunch of solutions with various trade-offs, pick the best strategy given various input constraints.
There are specialized tools out there that you can pay an arm and a leg for a license to do this, or you can have Claude one-off a project that gets the same result for $0.50 of AI credits. We live in an age of unprecedented intelligence abundance, and people are not used to this. I can have Claude implement something that would take a team of engineers months or years to do, and use it once then throw it away.
I say Claude specifically because in my experience none of the other models are really able to handle tasks like this.
Edit: an example prompt I put here: https://news.ycombinator.com/item?id=43639320
Talk is cheap. The bottom line is that I don't see any SAT solvers that you generated with Claude.
It’s not my job to make one for you.
Theorem proving.
What a fantastic article. One of the best on HN this year.
I first heard about Cyc as a child in a TV documentary about AI. The example involved a man shaving with an electric razor. Cyc concluded that while shaving the man was not human since the razor was an electrical device and electrical devices were not human. It really caught my imagination as a child and made me want to study AI. The magic diminished a bit once I learned more about how Cyc worked using prolog style relations and I ended up studying CS instead of AI but I still credit Cyc with sparking that initial interest in me.
Lenart seems like a strange man but we need obsessives and true believers like him to push through the various "winters". Who knows if knowledge graphs like Cyc will become relevant again in future as we seek to eliminate hallucinations from statistical learning.
I remember the very same example (re)-sparking my interest in AI back in the days.
My gut feeling says there is something in this approach that's needed to make GenAI work reliably. The brain has an associative feature, sure; but it's not very useful without filters sorting signal from nonsense, making sense of the content.
Have they been able to get Cyc to generate its own content in meaningful ways? I would expect such a system to eventually be able to derive a lot of details by itself, needing less and less spoon feeding.
> Cyc concluded that while shaving the man was not human since the razor was an electrical device and electrical devices were not human.
I can't parse this sentence? Is there supposed to be a comma before and after "while shaving"?
Yes. Sorry. I was actually just googling this and realised this same anecdote is in cited the intro to the Deep Learning book by Goodfellow et al. Their write-up is hopefully clearer:
"For example, Cyc failed to understand a story about a person named Fred shaving in the morning (Linde, 1992). Its inference engine detected an inconsistency in the story: it knew that people do not have electrical parts, but because Fred was holding an electric razor, it believed the entity “FredWhileShaving” contained electrical parts. It therefore asked whether Fred was still a person while he was shaving"
https://www.deeplearningbook.org/contents/intro.html
The (Linde, 1992) citation is they give is the 4th episode of a TV series - presumably the one I saw as a kid!
https://en.m.wikipedia.org/wiki/The_Machine_That_Changed_the...
And of course it's on YouTube:
https://youtube.com/clip/UgkxRcsHT-s1iZ-VRWFRXA-qg4kjTYe-a6j...
To be fair, the object "FredWhileShaving" is a state (of Fred), or simply an event, and as such it arguably is not a person in the first place. As persons are presumably neither states nor events.
Deleuze would argue something similar
along this line, “Gravity has no friends.”
http://blog.kenperlin.com/?p=2068
> seems like a strange man
seemed.
Unfortunately, he passed away a couple of years ago (which I wish I had known before now!)
You can still read his papers and textbooks. I thought he was an engaging speaker as well, and several of his lectures and interviews are available on youtube, as well as tributes and retrospectives.
A very interesting and worthwhile article (better than the comments here would suggest). However, I find it a bit of a pity that the author places so much emphasis on the assumption that the project has failed. The approach has not simply failed because the search for a solution has been going on for forty years. It took even more than forty years and costed at least as much before neural networks became really useful, and nobody would claim that the project failed because of that. And today's LLM are not really that intelligent yet. Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs. Perhaps then a greater benefit of this data will become apparent.
> Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs.
More likely, it will be made increasingly irrelevant as open alternatives to it are developed instead. The Wikipedia folks are working on some sort of openly developed interlingua that can be edited by humans, in order to populate Wikipedias in underrepresented languages with basic encyclopedic text. (Details very much TBD, but see https://en.wikipedia.org/wiki/Abstract_Wikipedia and https://meta.wikimedia.org/wiki/Abstract_Wikipedia ) This will probably be roughly as powerful as the system OP posits at some point in the article, that can generate text in both English and Japanese but only if fed with the right "common sense" to begin with. It's not clear exactly how useful logical inference on such statements might turn out to be, but the potential will definitely exist for something like that too, if it's found to be genuinely worthwhile in some way.
> made increasingly irrelevant as open alternatives to it are developed instead
Certainly interesting what these projects are going for, but it's unlikely an "open alternative", given that the degree of formalization and rigor achieved by Cyc's higher-order logic specification is likely not achievable by statistical learning, and a symbolic approach is barely achievable in a shorter time than Cyc.
It would be very surprising if the results from this approach were superior to simply machine-translating the entries from another language—because e.g. English already has so much content and contributor activity, and LLMs are already very good at translating. I can’t imagine you’d get more than a fraction of people’s interest in authoring entries in this abstract language.
LLMs are good at translating between languages that have significant amounts of written content on the internet. There are few languages in this category that do not already have correspondingly large Wikipedias.
There are plenty of languages with millions of speakers that are only rarely used in writing, often because some other language is enforced in education. If you try to use an LLM to translate into such a language, you'll just get garbage.
It's very easy for a hand-crafted template to beat an LLM if the LLM can't do the job at all.
https://www.wikidata.org/wiki/Wikidata:Main_Page, for those curious about the interlingua in question.
Strictly speaking, Wikidata is an existing project which only provides a rather restrictive model for its assertions; they are not fully compositional, thus are quite far from being able to express general encyclopedic text, especially in a way that can be 'seamlessly' translated to natural language. It does provide a likely foundation for these further planned developments, though.
> on the assumption the project has failed.
My daughter's PhD thesis was largely negative results. Even if the project had failed, we could learn from it if it wasn't so secretive. It could be much more open without being OSS!
It is at least as important to know which approaches do not work, but this gets significantly less press, which is not that attractive for scientists in the age of "publish or perish".
>> My daughter's PhD thesis was largely negative results.
Well, give us a link man! :)
You're forcing the proud dad function: https://pubmed.ncbi.nlm.nih.gov/36995257/
Absolutely. The proud nerd dad function :D
Congratulations to your daughter for her PhD. I'm guessing she has got it by now.
Sonic hedgehog signalling pathway! And what a date to submit a thesis.
Why is that a negative result, btw?
> Maybe Cycorp's knowledge base will be made generally accessible at some point
I would sooner hold my breath waiting for OpenAI to open up than Cycorp :)
> It took even more than forty years and costed at least as much before neural networks became really useful
The correct class of comparison to make with "neural networks" would be "symbolic AI" writ large. Symbolic AIs have been working quite well in some areas. Just not at all in terms of common sense reasoning, or anything approaching AGI.
If you want to keep "Cyc" in the comparison, then I would argue there is no comparison possible. Without exaggeration, there has never been a single project in AI as expensive as Cyc before 2020. Only with GPT-2 did the cost start to exceed the million USD mark. (Without exact figures, AlphaGo and Deep Blue probably also cost millions of dollars, but they unambiguously worked.)
It's also just not true that it took 40 years. Consider e.g. LeNet-5, which was up and running in 1998, and was used in ATMs to read real cheques. The main cost was 10 years of engineering stamina by LeCun's research group at Bell Labs. The finished version could be trained "for about 20 epoches over MNIST. It took 2 to 3 days of CPU time on a Silicon Graphics Origin 2000 server, using a single 200 MHz R10000 processor."
(1998 might technically be 40 years out from e.g. the inception of the perceptron in the 1950s, but if that is supposed to be our reference point for neural networks, then Cyc's reference point should be the inception of logical AIs in the same decade. And really, what use was Cyc in industry in 1998?)
Cyc is going great, according to the web site. "The Next Generation of Enterprise AI"[1]
Lenat himself died in 2023. Despite this, he is listed as the only member of the "leadership team".[2]
[1] https://cyc.com/
[2] https://cyc.com/leadership-team/
A sad epitaph
I’m very glad he tried, even if he only proved that it was impossible
He didn't prove any such thing.
Especially if you consider using a SOTA LLM to recreate and expand ResearchCyc (for fun and not profit)
Maybe Cyc was a success and Lenat lives on as it's consciousness?
Dead is dead. And even if Cyc had consciousness--which it doesn't--it certainly wouldn't have his consciousness.
It's an interesting thought experiment / philosophy / sci fi story premise though; if he spent all those years encoding his own thought processes and decision making into a program, would he have been able to create a convincing facsimile of himself and his "consciousness"? A turing test with a chatbot based on decades of self-reflection.
The 2014 film Transcendence touches upon this, but I can't recommend watching it. It's surface level and the rest of the film is mediocre.
Stephenson's "Fall; or, Dodge in Hell" deals with the hypothetical digitization of human minds. It has some interesting ideas like some of the synthetic minds deciding to wilfully become a hivemind
You're completely moving the goalposts. And I don't find it interesting at all (I mean, I find the general subject interesting and have delved into it for decades but I don't find this sort of casual question based on no such research, trying to connect it to the wrong thing [Cyc] at all interesting) ... would he have been able to create a convincing facsimile? If and only if the encoding method were effective--that's a tautology. Was Lenat's methodology effective to that end? No, of course not, and that wasn't its intent.
"based on decades of self-reflection"
Daniel Dennett--sadly lost to us--explained in detail why "self-reflection" is not even remotely effective to this end ... our internal processes are almost entirely inaccessible to us.
You can run a version of CYC that was released online as opencyc https://github.com/asanchez75/opencyc . This is when a version of the system was posted on source forge and the GitHub has the dataset and the KB and inference engine. Note it has been written in an old version of Java.
Many years ago, I went on a quest to read as much about AM and EURISKO as I could fine. Unfortunately, Lenat kept all of his work very private (secret, even), and there was very little to read other than high level papers and articles. I always felt that was a shame, as it prevented anyone else from building on anything Lenat built. I think we’re poorer for it.
You can run EURISKO yourself these days tho.
Too little, too late
The article is a very good review of Symbolic AI, in general, not just Cyc.
I have spent a lot of time with OpenCyc in the past, but haven’t touched it in ten years.
I believe tif there is a productive future for symbolic AI that it will involve using LLMs to construct knowledge graphs, symbolic relatikns, etc. from unstructured data.
>> The article is a very good review of Symbolic AI, in general, not just Cyc.
As a "review of symbolic AI" I found it uninformed and superficial and felt that it rehashed the same old points about how symbolic AI "failed", which are disputed by the facts; specifically the fact that major symbolic AI fields like SAT solving, automated theorem proving and planning and scheduling are still going strong and have produced real-world results, so much so that e.g. SAT solving, Planning, program verification, and automated theorem proving aren't even considered "AI" anymore because they now actually work, and work very well indeed.
Technically, you are probably correct. I did find it a good walk ‘down memory lane.’ I have been working in the field since 1982 and the article made me nostalgic.
With LLM itself using those graphs to validate its answers, etc. And at some point it will be internalized into architecture as a graph attention layer.
I would argue that Lenat was at least directionally correct in understanding that sheer volume of data (in Cyc's case, rules and facts) was the key in eventually achieving useful intelligence. I have to confess that I once criticized the Cyc project for creating an ever-larger pile of sh*t and expecting a pony to emerge, but that's sort of what has happened with LLMs.
I suspect at some point the pendulum will again swing back the other way and symbolic approaches will have some kind of breakthrough and become trendy again. And, I bet it will likely have something to do with accelerating these systems with hardware, much like GPUs have done for neural networks, in order to crunch really large quantities of facts
The Bitter Lesson has a few things to say about this.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
The Bitter Lesson says "general methods that leverage computation are ultimately the most effective". That doesn't seem to rule out symbolic approaches. It does rule out anything which relies on having humans in the loop, because terabytes of data plus a dumb learning process works better than megabytes of data plus expert instruction.
(I know your message wasn't claiming that The Bitter Lesson was explicitly a counterpoint, I just thought it was interesting.)
Imho, this is wrong. Even independent of access to vast amounts of compute, symbolic methods seem to consistently underperform statistical/numerical ones across a wide variety of domains. I can't help but think that there's more to it than just brute force.
I've lost count how many times I've written the same words in this thread but: SAT Solving, Automated Theorem Proving, Program Verification and Model Checking, Planning and Scheduling. These are not domains where symbolic methods "consistently underperform" anything.
You guys really need to look into what's been going on in classical AI in the last 20-30 years. There are two large conferences that are mainly about symbolic AI, IJCAI and AAAI. Then there's all the individual conferences on the above sub-fields, like the International Conference on Automated Planning and Scheduling (ICAPS). Don't expect to hear about symbolic AI on social media or press releases from Alpha and Meta, but there's plenty of material online if you're interested.
Real AGI will need a way to reason about factual knowledge. An ontology is a useful framework for establishing facts without inferring them from messy human language.
These guys are trying to combine symbolic reasoning with LLMs somehow: https://www.symbolica.ai/
check out Imandra's platform for neurosymbolic AI - https://www.imandra.ai/
Or maybe program synthesis combined by LLMs might be the way?
It does seem like the Cyc people hit the wall with simply collecting facts. Having to have a human in the loop.
The problem I think is if you have LLMs figuring out the propositions, the whole system is just as prone to garbage-in-garbage-out as LLMs are.
But
a) The pile of LLM training data is vastly larger. b) The data is actual human utterances in situ--these are ponies, not pony shit. c) LLMs have no intelligence ... they channel the intelligence of a vast number of humans by pattern matching their utterances to a query. This has indeed proved useful because of how extremely well the statistical apparatus works, but the fact that LLMs have no cognitive states puts great limits on what this technology can achieve.
With Cyc, OTOH, it's not even clear what you can get out of it. The thing may well be useful if combined with LLMs, but it's under lock and key.
The big conclusions about symbolic AI that the author reaches based on this one system and approach are unwarranted. As he himself notes, "Even Ernest Davis and Gary Marcus, highly sympathetic to the symbolic approach to AI, found little evidence for the success of Cyc, not because Cyc had provably failed, but simply because there was too little evidence in any direction, success or failure."
>> they channel the intelligence of a vast number of humans by pattern matching their utterances to a query.
Just a little problem with that: to understand the utterances of a vast number of humans you need to channel it to something that can understand the utterances of humans in the first place. Just channeling it around from statistic to statistic doesn't do the trick.
Um, the "something" is the person reading the LLM's output. I'm afraid that you have completely missed the context and point of the discussion, which was not about LLMs understanding things--they understand nothing ("LLMs have no cognitive states"). But again, "because of how extremely well the statistical apparatus works", their outputs are useful to intelligent consumers who do have cognitive states--us.
The dataset for lots of LLMs is literally called "The Pile" lmao
That’s hilarious, but at least Llama was trained on libgen, an archive of most books and publications by humanity, no? Except for the ones which were not digitized I guess
So there is probably a big pile of Reddit comments, twitter messages, and libgen and arxiv PDFs I imagine
So there is some shit, but also painstakingly encoded knowledge (ie writing), and yeah it is miraculous that LLMs are right as often as they are
libgen is far from an archive of "most" books and publications, not even close.
The most recent numbers from libgen itself are 2.4 million non-fiction books and 80 million science journal articles. The Atlantic's database published in 2025 has 7.5 million books.[0] The publishing industry estimates that many books are published each year. As of 2010, Google counted over 129 million books[1]. At best an LLM like Llama will have have 20% of all books in its training set.
0. https://www.theatlantic.com/technology/archive/2025/03/libge...
1. https://booksearch.blogspot.com/2010/08/books-of-world-stand...
On libgen.mx they claim to have 33,569,200 books and 84,844,242 articles
Still an order of magnitude short of "all", and falling farther behind every year.
It's a miracle, but it's all thanks to the post-training. When you think of it, for so-called "next token predictors", LLMs talk in a way that almost no one actually talks, with perfect spelling and use of punctuation. The post-training somehow is able to get them to predict something along the lines of what a reasonably intelligent assistant with perfect grammar would say. LLMs are probably smarter than is exposed through their chat interface, since it's unlikely the post-training process is able to get them to impersonate the smartest character they'd be capable of impersonating.
I dunno I actually think say Claude AI SOUNDS smarter than it is, right now
It has a phenomenal recall. I just asked it about "SmartOS", something I knew about, vaguely, in ~2012, and it gave me a pretty darn good answer. On that particular subject, I think it probably gave a better answer than anyone I could e-mail, call, or text right now
It was significantly more informative than wikipedia - https://en.wikipedia.org/wiki/SmartOS
But I still find it easy to stump it and get it to hallucinate, which makes it seem dumb
It is like a person with good manners, and a lot of memory, and which is quite good at comparisons (although you have to verify, which is usually fine)
But I would not say it is "smart" at coming up with new ideas or anything
I do think a key point is that a "text calculator" is doing a lot of work ... i.e. summarization and comparison are extremely useful things. They can accelerate thinking
https://ai-2027.com/ postulates that a good enough LLM will rewrite itself using rules and facts... sci-fi, but so is chatting with a matrix multiplication.
I doubt it. The human mind is a probabilistic computer, at every level. There’s no set definition for what a chair is. It’s fuzzy. Some things are obviously in the category, and some are at the periphery of it. (Eg is a stool a chair? Is a log next to a campfire a chair? How about a tree stump in the woods? Etc). This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
> There’s no set definition for what a chair is.
Sure there is: a chair is anything upon which I can comfortably sit without breaking it.
I find this very amusing. In philosophy of science some 20+ years ago I had a wonderful prof who went through 3(?) periods of thought. He laid out this argument, followed by the arguments seen below in this thread in various ways, in a systematic way where he convinced you that one way of thinking was correct, you took the midterm, then the next day he would lead with "everything you know is wrong, here's why.". It was beautiful.
He noted that this evolution of thought continued on until people generally argued that concepts/definitions that let you do meaningful things (your definition of meaningful, doesn't really matter what it is), are the way to go. The punchline at the very end, which happened to be the last thing I regurgitated on my last undergraduate exam, was him saying something along the lines of "Science, it beats hanging out in malls."
All this to say that if we read a little philosophy of science, that was done a long time ago (way before the class I took), things would make more sense.
I have definitely broken chairs upon sitting in them, which someone else could have sat in just fine. So it's unclear why something particular to me would change the chair-ness of an object.
Similarly, I've sat in some very uncomfortable chairs. In fact, I'd say the average chair is not a particularly comfortable one.
For a micro-moment before giving in it was a chair, then it broke. Now its no longer a chair. Its a broken chair.
That's not one, but two particularities that aren't latent to the chair itself: me (the sitter), and time.
Do you really have a personal ontology that requires you to ask the tense and person acting on a thing to know what that thing is? I suspect you don't; most people don't, because it would imply that the chair wouldn't be a chair if nobody sat on it.
A stump isn't a chair until someone decides to sit on it, at that point it becomes chair _to_ that person. Chair is only capable of acting as "chair" object if constraints are met in regards to sitter.
This is very complicated, because it now implies:
1. I can intend to sit on a chair but fail, in which case it isn't a chair (and I didn't intend to sit on it?)
2. I can intend to have my dog sit on my chair, but my dog isn't a person and so my chair isn't a chair.
This is-use distinction you're making is fine; most people have an intuition that things "act" as a thing in relation to how they're used. But to take it a step forwards and claim that a thing isn't its nature until a person sublimates their intent towards it is very unintuitive!
(In my mind, the answer is a lot simpler: a stump isn't a chair, but it's in the family network of things that are sittable, just like chairs and horses. Or to borrow Wittgenstein, a stump bears a family resemblance to a chair.)
I'm the person who asked about the definition of a chair up thread.
Just to make a very obvious point: Nobody thinks of the definition for a chair as a particularly controversial idea. But clearly:
- We don't all agree on what a chair is (is a stump a chair or not?).
- Nobody in this thread has been able to give a widely accepted definition of the word "chair"
- It seems like we can't even agree on what criteria are admissible in the definition. (Eg, does it matter that I can sit on it? Does it matter that I can intend to sit on it? Does it matter that my dog can sit on it?)
If even defining what the word "chair" means is beyond us, I hold little hope that we can ever manually explain the concept to a computer. Returning to my original point above, this is why I think expert systems style approaches are a dead end. Likewise, I think any AI system that uses formal or symbolic logic in its internal definitions will always be limited in its capacity.
And yet, I suspect chatgpt will understand all of the nuance in this conversation just fine. Like everyone else, I'm surprised how "smart" transformer based neural nets have become. But if anything has a hope of achieving AGI, I'm not surprised that:
- Its something that uses a fuzzy, non-symbolic logic internally.
- The "internal language" for its own thoughts is an emergent result of the training process rather than being explicitly and manually programmed in.
- That it translates its internal language of thought into words at the end of the thinking / inference process. Because - as this "chair" example shows - our internal definition for what a chair is is seems clear to us. But it doesn't necessarily mean we can translate that internal definition into a symbolic definition (ie with words).
I'm not convinced that current transformer architectures will get us all the way to AGI / ASI. But I think that to have a hope of achieving human level AI, you'll always want to build a system which has those elements of thought. Cyc, as far as I can tell, does not. So of course, I'm not at all surprised its being dumped.
What if it breaks in a way which renders it no longer a chair for you but not others?
This seems to imply that what is or is not a chair is a subjective or conditional.
A broken chair is by definition a chair. You just said it!
> Sure there is: a chair is anything upon which I can comfortably sit without breaking it.
« It is often said that a disproportionate obsession with purely academic or abstract matters indicates a retreat from the problems of real life. However, most of the people engaged in such matters say that this attitude is based on three things: ignorance, stupidity, and nothing else.
Philosophers, for example, argue that they are very much concerned with the problems posed by real life.
Like, for instance, “what do we mean by real?”, and “how can we reach an empirical definition of life?”, and so on.
One definition of life, albeit not a particularly useful one, might run something like this: “Life is that property which a being will lose as a result of falling out of a cold and mysterious cave thirteen miles above ground level.”
This is not a useful definition, (A) because it could equally well refer to the subject’s glasses if he happens to be wearing them, and (B) because it fails to take into account the possibility that the subject might happen to fall onto, say, the back of an extremely large passing bird.
The first of these flaws is due to sloppy thinking, but the second is understandable, because the mere idea is quite clearly, utterly ludicrous. »
— Douglas Adams
So a warm and smelly compost pile is a chair? A cold metal park bench is not a chair (because it's uncomfortable)?
A beanbag is a chair? Perhaps a chair should be something on which one can comfortably sit without breaking that has a back and four legs. I suppose then a horse would be a chair.
Would a chair on the Moon (or somewhere else inaccessible to you) be a chair?
What if there was a chair in the forest and no-one there to sit on it?
Is a mountain a chair?
With the right attitude (and perhaps altitude), yes :)
My bicycle is a chair, apparently.
So you're just ignoring all the probabilistic, fuzzy etc. Prologs etc. which do precisely that? https://github.com/lab-v2/pyreason
> Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
The counterposition to this is no more convincing: cognition is fuzzy, but it's not really clear at all that it's probabilistic: I don't look at a stump and ascertain its chairness with a confidence of 85%, for example. The actual meta-cognition of "can I sit on this thing" is more like "it looks sittable, and I can try to sit on it, but if it feels unstable then I shouldn't sit on it." In other words, a defeasible inference.
(There's an entire branch of symbolic logic that models fuzziness without probability: non-monotonic logic[1]. I don't think these get us to AGI either.)
[1]: https://en.wikipedia.org/wiki/Non-monotonic_logic
Which word will I pick next in this sentence? Is it deterministic? I probably wouldn’t respond the same way if I wrote this comment in a different mood, or at a different time of day.
What I say is clearly not deterministic for you. You don’t know which word will come next. You have a probability distribution but that’s it. Banana.
I caught a plane yesterday. I knew there would be a plane (since I booked it) and I knew where it would go. Well, except it wasn’t certain. The flight could have been delayed or cancelled. I guess I knew there would be a plane with 90% certainty. I knew the plane would actually fly to my destination with a 98% certainty or something. (There could have been a malfunction midair). But the probability I made it home on time rose significantly when I saw the flight listed, on time, at the airport.
Who I sat next to was far less certain - I ended up sitting next to a 30 year old electrician with a sore neck.
My point is that there is so much reasoning we do all the time that is probabilistic in nature. We don’t even think about it. Other people in this thread are even talking about chairs breaking when you sit on them - every time you sit on a chair there’s a probability calculation you do to decide if the chair is safe, and will support your weight. This is all automatic.
Simple “fuzzy logic” isn’t enough because so many probabilities change as a result of other events. (If the plane is listed on the departures board, the prediction goes up!). All this needs to be modelled by our brains to reason in the world. And we make these calculations constantly with our subconscious. When you walk down the street, you notice who looks dangerous, who is likely to try and interact with you, and all sorts of things.
I think that expert systems - even with some fuzzy logic - are a bad approach because systems never capture all of this reasoning. It’s everywhere all the time. I’m typing on my phone. What is the chance I miss a letter? What is the chance autocorrect fixes each mistake I make? And so on, constantly and forever. Examples are everywhere.
To be clear, I agree that this is why expert systems fail. My point was only that non-monotonic logics and probability have equal explanatory power when it comes to unpredictability: the latter models with probability, and the former models with relations and defeasible defaults.
This is why I say the meta-cognitive explanation is important: I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case. Instead, when people use words like “likely” and “unlikely,” they’re typically expressing a defeasible statement (“typically, a stranger who approaches me on the street is going to ask me for money, but if they’re wearing a suit they’re typically a Jehovah’s Witness instead”).
> I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case.
Interesting. I don't think I agree.
I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets. For example, if someone (with no special knowledge) offered a 50/50 bet that your dining chair will break next time you sit on it, well, that sounds like a safe bet! Easy money! What about if the odds changed - so, if it breaks you give them $60, and if it doesn't break they give you $40? I'd still take that bet. What about 100-1 odds? 1000-1? There's some point where you start to say "no no, I don't want to take that bet." or even "I'd take that bet if we swap sides".
Somewhere in our minds, we hold an intuition around the probability of different events. But I think it takes a bit of work to turn that intuition into a specific number. We use that intuition for a lot of things - like, to calibrate how much surprise we feel when our expectation is violated. And to intuitively decide how much we should think through all the alternatives. If we place a bet on a coin flip, I'll think through what happens if the coin comes up heads or if it comes up tails. But if I walk into the kitchen, I don't think about the case that I accidentally stub my toe. My intuition assigns that a low enough probability that I don't think about it.
Talking about defeasible statements only scratches the surface of how complex our conditional probability reasoning is. In one sense, a transformer model is just that - an entire transformer based LLM is just a conditional probability reasoning system. The entire model of however many billions of parameters is all a big conditional probability reasoning machine who's only task is to figure out the probability distribution over the subsequent token in a stream. And 100bn parameter models are clearly still too small to hit the sweet spot. They keep getting smarter as we add more tokens. If you torture an LLM model a little, you can even get it to spit out exact probability predictions. Just like our human minds.
I think these kind of expert systems fail because they can't do the complex probability reasoning that transformer models do. (And if they could, it would be impossible to manually write out the - perhaps billions - of rules it would need to accurately reason about the world like chatgpt can.)
> I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets.
I think these are different things! I can definitely make myself think about probabilities, but that's a cognitive operation rather than a meta-cognitive one.
In other words: I think what you're describing as "a bit of work" around intuitions is our rationalization (i.e., quantification) of an process that's internally non-statistical, but defeasible instead. Defeasibility relationships can have priorities and staggerings, which we turn into fuzzy likelihoods when we express them.
My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression. I could make one up, but as you note with torturing the LLM, I'm doing it to "make progress," not because it's a true statement of probability.
(I think expert systems fail for a reason that's essentially not about probability reasoning, but dimensionality -- as the article mentions Cyc has at least 12 dimensions, but there's no reason to believe our thoughts have only or exactly these 12. There's also no reason to believe we can ever model the number of dimensions needed, given that adding dimensions to an encoded relation set is brutally exponential.)
>My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression.
LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
> LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
(Humans are imprecise, LLMs are imprecise, thermometers are imprecise, but don't stick me or my computer in an oven, please.)
>Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
Doesn't have to be like an LLM's to be probabilistic
>I don't look at a stump and ascertain its chairness with a confidence of 85%
But i think you did. Not consciously, but i think your brain definitely did.
https://www.nature.com/articles/415429a https://pubmed.ncbi.nlm.nih.gov/8891655/
These papers don't appear to say that: the first one describes the behavior as statistically optimal, which is exactly what you'd expect for a sound set of defeasible relations.
Or intuitively: my ability to determine whether a bird flies or not is definitely going to be statistically optimal, but my underlying cognitive process is not itself inherently statistical: I could be looking at a penguin and remembering that birds fly by default except when they're penguins, and only then if the penguin isn't wearing a jetpack. That's a non-statistical set of relations, but its external observation is modeled statistically.
>which is exactly what you'd expect for a sound set of defeasible relations.
This is a leap. While a complex system of rules might coincidentally produce behavior that looks statistically optimal in some scenarios, the paper (Ernst & Banks) argues that the mechanism itself operates according to statistical principles (MLE), not just that the outcome happens to look that way.
Moreover, it's highly unlikely, bordering on impossible, to reduce the situations the brain deals with even on a daily basis into a set of defeasible statements.
Example: Recognizing a "Dog"
Defeasible Attempt: is_dog(X) :- has_four_legs(X), has_tail(X), barks(X), not is_cat(X), not is_fox(X), not is_robot_dog(X).
is_dog(X) :- has_four_legs(X), wags_tail(X), is_friendly_to_humans(X), not is_wolf(X).
How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
Ultimately, the proof as they say, is in the pudding. By the way, the CyC we are all talking about here is non-monotonic. https://www.cyc.com/wp-content/uploads/2019/07/First-Orderiz...
If you've tried something for decades and it's not working, and it doesn't even look like it's working and experiments with the brain suggest probabilistic inference and probabilistic inference machines work much better than the alternatives ever did, you have to face the music.
> How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
This is the dimensionality mentioned in the adjacent post, and it's true of a probabilistic approach as well: an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
(I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.)
>This is the dimensionality mentioned in the adjacent post
LLMs are only a few years old but symbolic ai was abandoned for NLP, Computer Vision etc long before that. Why ? Because the alternative was just that bad and more importantly, never seemed to really budge with effort. Companies didn't wake up one morning and pour hundreds of millions into LMs. In fact, NNs were the underdog for a very long time. They poured more and more money into it because it got better and better with investment.
There is zero reason to think even more dimensionality would do anything but waste even more time. At least the NN scalers can look back and see it work in the past. You don't even have that.
>an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
It doesn't take much to train LLMs to 'give up'. Open AI talk about this from time to time. It's just not very useful with a tendency to overcorrect. And humans hallucinate (and otherwise have weird failure modes) all the time. We just call them funny names like dunning kruger and optical illusions. Certainly less than current SOTA LLMs but it happens all the same.
>I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.
The point is not about Cyc. They're hardly the only attempt at non-monotonic logic. The point is that they should work much better than they do if there's anything to it. Again, forget recent LLMs. Even when we were doing 'humble' things like spelling error detection & correction, text compressors, voice transcription boosters, embeddings for information retrieval, recommenders, knowledge graph creation (ironically enough), machine translation services, etc these systems were not even in the conversation. They performed that poorly.
I think we're talking past each other. I'm not interested in defending symbolic AI at all; it's clear it's failed. All told, I would not say I'm particularly interested in any kind of AI.
I'm interested in theory of mind, and I think defeasibility with a huge number of dimensions is a stronger explanation of how humans behave and think than something resembling an LLM. This doesn't somehow mean that LLMs haven't "won" (they have); I just don't think they're winning at human-like cognition. This in turn does not mean we could build a better alternative, either.
> The human mind is a probabilistic computer, at every level.
Fair enough, but an airplane's wing is not very similar to a bird's wing.
That argument would hold a lot more weight if Cyc could fly. But as this article points out, decades of work and millions of dollars have utterly failed to get it off the ground.
right, but as others have pointed out the amount of money invested in Cyc is approx 2 orders of magnitude less than what was invested in LLMs. So maybe the method was OK, but it was insufficiently resourced.
Maybe, but I doubt it. Transformer models got all that investment because small transformers work and large transformers work better than small transformer models. We can extend the line and predict what happens when the model is scaled.
Small Cyc doesn't do anything useful. Large Cyc doesn't do anything useful. Why should we make huge Cyc? If we extend the line out, all evidence predicts that it'll still be useless.
> There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
Are you sure? In terms of theoretical foundations for AGI, AIXI is probabilistic but godel-machines are proof based and I think they'd meet criteria for deterministic / symbolic. Non-monotonic and temporal logics also exist, where chairness exists as a concept that might be revoked if 2 or more legs are missing. If you really want to get technical then by allowing logics with continuous time and changing discrete truth values, then you can probably manufacture a fuzzy logic where time isn't considered but truth/certainty values are continuous. Your ideas about logic might be too simple, it's more than just Aristotle
Not person you are replying to, just FYI.
I don't know, it all seems like language games to me. The meaning is never in its grammar, but in its usage. The usage is arbitrary and capricious. I've not discovered how more nuanced forms of logics have ever really grappled with this.
In the 1980s, we used to talk about the silliness of the “grandmother neuron” - the idea that one neuron would capture an important thing, rather than a distributed representation.
> This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition.
That is indeed true. But we do have classic fuzzy logic, and it can be used to answer these questions. E.g. a "stool" maybe a "chair", but "automobile" is definitely not.
Maybe the symbolic logic approach could work if it's connected with ML? Maybe we can use a neural network to plot a path in the sea of assertions? Cyc really seems like something that can benefit the world if it's made open under some reasonable conditions.
> That is indeed true. But we do have classic fuzzy logic, and it can be used to answer these questions. E.g. a "stool" maybe a "chair", but "automobile" is definitely not.
I’m not convinced that classical fuzzy logic will ever solve this - at least not if every concept needs to be explicitly programmed in. What a “chair” is sort of subtly changes at a furniture store and at a campsite. Are you going to have someone explicitly, manually program all of those nuances in? No way! And without that subtlety, you aren’t going to end up with a system that’s as smart as chatgpt. Challenge me on this if you like, but we can play this game with just about any word you can name - more or less everything except for pure mathematics.
And by the way, modern ML approaches understand all of those nuances just fine. It’s not clear to me what value - if any - symbolic logic / expert systems provide that chatgpt isn’t perfectly capable of learning on its own already.
"The human mind is a probabilistic computer, at every level."
We don't know that. It's mostly probabilistic. That innate behavior exists suggests some parts might be deterministic.
Words are used due to the absence of things. They fill an immediate experiential void and stand in for something else, because you want or need another person to evoke some fantasy to fill this absence and make understanding possible.
If you have a mind and it is a computer, then it is because of nurture, because the brain is nothing like a computer, and computers simulating language are nothing like brains.
That is not what is suggested. Llm still fuzzy mess, but supervisor / self editing is rules based
The way I see it:
(1) There is kind of a definition of a chair. But it's very long. Like, extremely long, and includes maybe even millions to billions of logical expressions, assuming your definition might need to use visual or geometric features of a given object to be classified as a chair (or not chair).
This is a kind of unification of neural networks (in particular LLMs) and symbolic thought: large enough symbolic thought can simulate NNs and vice versa. Indeed even the fact that NNs are soft and fuzzy does not matter theoretically, it's easy to show logical circuits can simulate soft and fuzzy boundaries (in fact, that's how NNs are implemented in real hardware! as binary logic circuits). But I think specific problems have varying degrees of more natural formulation as arithmetic, probabilistic, linear or fuzzy logic, on one hand, and binary, boolean-like logic on the other. Or natural formulations could involve arbitrary mixes of them.
(2) As humans, the actual definitions (although they may be said to exist in a certain way at a given time[1]) vary with time. We can, and do, invent new stuff all the time, and often extend or reuse old concepts. For example, I believe the word 'plug' in english likely well predates modern age, probably used to refer to original electrical power connectors. Nowadays there are USB plugs, which may not carry power at all, or audio plugs, etc. (maybe there are better examples). In any case the pioneer(s) usually did not envision all a name could be used for, and uses evolve.
(3) Words are used as tools to allow communication and, crucially, thought. There comes a need to put a fence (or maybe a mark) in abstract conceptual and logic space, and we associate that with a word. Really a word could be "anything we want to communicate", represent anything. In particular changes to the states of our minds, and states themselves. That's usually too general, most words are probably nouns which represent classifications of objects that exist in the world (like the mentioned chair) -- the 'mind state' definition is probably general enough to cover words like 'sadness', 'amazement', etc., and 'mind state transitions' probably can account for everything else.
We use words (and associated concepts) to dramatically reduce the complexity of the world to enable or improve planning. We can then simplify our tasks into a vastly simpler logical plan: even something simple like put shoes, open door, go outside, take train, get to work -- without segmenting the world into things and concepts (it's hard to even imagine thought without using concepts at all -- it probably happens instinctively), the number of possibilities involved in planning and acting would be overwhelming.
Obligatory article about this: https://slatestarcodex.com/2014/11/21/the-categories-were-ma...
---
Now this puts into perspective the work of formalizing things, in particular concepts. If you're formalizing concepts to create a system like Cyc, and expect it to be cheap, simple, reliable, and function well into the future, by our observations that should fail. However, formalization is still possible, even if expensive, complex, and possibly ever changing.
There are still reasons you may want to formalize things, in particular to acquire a deeper understanding of those things, or when you're okay in creating definitions set in stone because they will be confined to a group being attentive and restrictive to their formal definitions (and not, as natural language, evolving organically according to convenience): that's the case with mathematics. The peano axioms still define the same natural numbers; and although names may be reused, you can usually specify them to a particular axiomatic definition that will never change. And thus we can keep building facts on those foundations forever -- while what is a 'plug' in natural language might change (and associated facts about plugs become invalid), we can define mathematical objects (like 'natural numbers') with unchanging properties, and ever-valid and potentially ever-growing valid facts to be known about them, reliably. So fixing concepts in stone more or less (at least when it comes to a particular axiomatization) is not such a foolish endeavor it may look like, quite the opposite! Science in general benefits from those solid foundations.
I think eventually even some concepts related to human emotions and specially ethics will be (with varying degrees of rigor) formalized to be better understood. Which doesn't mean human language should (or will) stop evolving and being fuzzy, it can do so independently of formal more rigid counterparts. Both aspects are useful.
[1] In the sense that, at a given time, you could (theoretically) spend an enormous effort to arrive at a giant rule system that would probably satisfy most people, and most objects referred to as chairs, at a given fixed time.
[dead]
How will the rules and facts be connected? By some discrete relationship? This stuff only works for math, and is the basis for the bitter lesson.
Intelligence is compression, and this is the opposite of that
A similar failure of GOFAI was ABBYY's (a Russian company which was for a long time market leader in OCR software) monumental (multi decade long) attempt of creating advanced translation software entirely based on complex formal grammar parsing.
The story behind it is really interesting. This article was written by someone who worked at ABBYY:
https://sysblok.ru/blog/gorkij-urok-abbyy-kak-lingvisty-proi...
The piece is in Russian but can (ironically) be read in good English by using e.g. the Google Translate feature inside Chrome. Which is of course entirely based on machine learning.
The story is essentially similar to Cyc: symbolic AI/logical AI/GOFAI can produce initially impressive results (ABBYY was way better than early forms of Google Translate), but symbolic approaches doesn't scale well. Big Data + machine learning wins out eventually. The piece above mentions a 2009 piece from Google which put forward this thesis. "The Unreasonable Effectiveness of Data":
https://static.googleusercontent.com/media/research.google.c...
Note that 2009 was significantly before the existence of large language models, transformers, or even AlexNet.
A big Cyc thread about a year ago https://news.ycombinator.com/item?id=40069298
A former employee of Cyc did an insightful AMA on HN back in 2019: https://news.ycombinator.com/item?id=21783828
> But the longer I worked there the more I felt like the plan was basically:
> 1. Manually add more and more common-sense knowledge and extend the inference engine
> 2. ???
> 3. AGI!
In retrospect, this reasoning doesn't seem to be so wrong.
I mean if I were to oversimplify and over-abstract AGI into a long list of if / elses, that's how I'd go about it. It's just that there's A Lot to consider.
I don't have strong opinions about Cyc in either direction, but this obit feels pretty mean-spirited to me: it imputes moral failure on Lenat and others for working on symbolic AI for 40 years, without acknowledging that Lenat's basic belief ("priming the pump") is shared by ML too -- the main difference is that ML needed the pump to be primed with an overwhelming amount of compute, which grew faster than Cyc's ability to ingest facts and relations.
(This isn't to imply that I think symbolic AI "works"; only that more perspective on underlying beliefs is due.)
I don't think there's anything wrong with exploring a field for decades. There are many scientists who have a mix of successes and failures. But this guy spend his whole life and many years of other people's lives trying one single thing that never really worked. You could call that being a single-minded visionary, but I don't think it's unreasonable for others to think it either kooky or a giant waste.
A useful comparison to me here is all the alchemical efforts to turn lead into gold. Can modern physicists do that? Not economically, but sure. [1] If alchemists had just persisted, would they have gotten there too? No, it was a giant waste, and pretty loony to a modern eye. And I'd say both alchemists and a number of AI proponents both are so wrapped up in pursuing specific outcomes (gold, AGI) that they indulge in a lot of magical thinking.
[1] https://www.scientificamerican.com/article/fact-or-fiction-l...
I don't think this is correct. The capability for drawing useful logical inferences from something like a Cyc knowledge base is far more compute limited than just doing ML on any given amount of data. We're talking exponential (or worse!) vs. pure linear scaling. This is the real-world, practical reason why the Cyc folks eventually found no value at all in their most general inference engine, and ended up exclusively relying on their custom-authored, more constrained inference generators instead.
Again, I'm not saying Cyc's approach is correct. I'm saying that the underlying hope that made Lenat plow through the AI winter is the same one that made ML researchers plow through it. It's just that the ML researchers reached the end of the tunnel first (for some senses of first).
Symbolic AI works. So much that it's "not AI" anymore.
In spite of all of this, Cycorp is still in business, and have pivoted to healthcare automation, including apparently insurance denials. I wonder if the full Cyc knowledg base will ever end up being released to the public domain, or whether it will simply fade away into nonexistence as proprietary data?
> Cycorp is still in business, and have pivoted to healthcare automation, including apparently insurance denials.
There's a lesson here for AI entrepreneurs: If it's hard to make systems that give actually good, actionable advice, you can always pivot to authoritatively telling people what they want to be told instead. It's probably a bigger market anyway. Gordon Way would be proud.
Your tongue is in your cheek, I think, but go and read any discussion forum populated by "big 4" consultants and you'll see a high degree of worry that they are being replaced by AI. So yes, LLMs are indeed great at "authoritatively telling people what they want to be told", and they're already being used that way!
I hope it doesn’t die in obscurity. Surely, it’s still valuable even if it failed to deliver on its original promises.
> Cyc grew to contain approximately 30 million assertions at a cost of $200 million and 2,000 person-years. Yet despite Lenat’s repeated predictions of imminent breakthrough, it never came.
That seems like pretty small potatoes compared to how much has been spent on LLMs these days.
Or to put it another way: if global funding for LLM development had been capped at $200m, how many of them would even exist?
Language models repeatedly delivered practical, real-world economic value at every step of the way from at least n-grams on. (Remember the original 'unreasonable effectiveness of data'?) The applications were humble and weren't like "write all my code for me and then immanentize the eschaton", but they were real things like spelling error detection & correction, text compressors, voice transcription boosters, embeddings for information retrieval, recommenders, knowledge graph creation (ironically enough), machine translation services, etc. In contrast, Yuxi goes through the handful of described Cyc use-cases from their entire history, and it's not impressive.
> Remember the original 'unreasonable effectiveness of data'?
That came out in 2009, correct? I wonder how much was spent on LLMs up to that point.
> In contrast, Yuxi goes through the handful of described Cyc use-cases from their entire history, and it's not impressive.
They're also not humble. Maintain a semantic database of terrorist cells? Create a tutoring AI? These seem closer to the things that LLMs are currently being used for, with middling success, after vastly more money has been pumped into the field.
Whereas most of the uses you describe for early LLMs are far more humble (spelling error detection & correction, text compressors), and also a lot more successful.
Which makes me think that CYC went first for the big targets, and fell on its face, rather than spending a few decades building up more modest accomplishments. In hindsight that would have obviously been a much better strategy, but honestly—it feels like that would have been an obviously better strategy in non-hindsight as well. I don't know why CYC went that way.
> That came out in 2009, correct? I wonder how much was spent on LLMs up to that point.
Quite a lot. Look back at the size of the teams working on language models at IBM, Microsoft, Google, etc, and think about all the decades of language model research going back to Shannon and quantifying the entropy of English. Or the costs to produce the datasets like the Brown Corpus which were so critical. And keep in mind that a lot of the research and work is not public for language models; stuff like NSA interest is obvious, but do you know what Bob Mercer did before he vanished into the black hole of Renaissance Capital? I recently learned from a great talk (spotted by OP, as it happens) https://gwern.net/doc/psychology/linguistics/bilingual/2013-... that it was language modeling!
I can't give you an exact number, of course, but when you consider the fully-loaded costs of researchers at somewhere like IBM/MS/G is usually at least several hundred thousand dollars a year and how many decades and how many authors there are on papers and how many man-years must've been spent on now-forgotten projects in the 80s and 90s to scale to billions of word corpuses to train the n-gram language models (sometimes requiring clusters), I would have to guess it's at least hundreds of millions cumulative.
> They're also not humble.
Funnily enough, the more grandiose use-cases of LMs actually were envisioned all the way back at the beginning! In fact, there's an incredible science fiction story you've never heard of which takes language models, quite literally, as the route to a Singularity, from 1943. You really have to read it to believe it: "Fifty Million Monkeys", Jones 1943 https://gwern.net/doc/fiction/science-fiction/1943-jones.pdf
> I don't know why CYC went that way.
If you read the whole OP, which I acknowledge is quite a time investment, I think Yuxi makes a good case for why Lenat culturally aimed for the 'boil the ocean' approach and how they refused to do more incremental small easily-benchmarked applications as distractions and encouraging deeply flawed paradigms and how they could maintain it for so long. (Which shouldn't be too much of a surprise. Look how much traction DL critics on HN get, even now.)
> Quite a lot. Look back at the size of the teams working on language models at IBM, Microsoft, Google, etc, and think about all the decades of language model research going back to Shannon and quantifying the entropy of English.
I wonder at what point the money spent on LLMs matched the $200 million that was ultimately spent on CYC.
> Funnily enough, the more grandiose use-cases of LMs actually were envisioned all the way back at the beginning!
Oh, I know—but those grandiose use cases still have yet to materialize, despite the time and money spent. But the smaller scale use cases have borne fruit.
> there's an incredible science fiction story you've never heard of which takes language models, quite literally, as the route to a Singularity, from 1943. You really have to read it to believe it: "Fifty Million Monkeys", Jones 1943
Thanks, I'll read that.
> If you read the whole OP, which I acknowledge is quite a time investment, I think Yuxi makes a good case for why Lenat culturally aimed for the 'boil the ocean' approach and how they refused to do more incremental small easily-benchmarked applications as distractions and encouraging deeply flawed paradigms and how they could maintain it for so long.
I read it for a chunk, but yeah, not the whole way.
> That seems like pretty small potatoes compared to how much has been spent on LLMs these days.
It seems to be a pretty high cost, at more than $6 per assertion. Wikidata - the closest thing we have to a "backbone for the Semantic Web" right now - contains around 1.6G bare assertions describing 115M real-world entities, and that's a purely volunteer project.
Global funding would never have been capped at $200M for LMs because they were obviously useful from the get go and only got more useful with more investment.
Forget CYC, Forget LLMs. We abandoned Symbolic-AI for Neural Networks in NLP long before the advent of the science-fiction esque transformer LLMs. That's how terrible they were.
It wasn't for a lack of trying either. NNs were the underdogs. Some of the greatest minds desparately wanted the symbolic approach to be a valid one and tried for literally decades, and while I wouldn't call it a 'failure', it just couldn't handle anything fuzzy without a rigidly defined problem space, which is kind of unfortunate seeing as that is the exact kind of intelligence that actually exists in the real world.
It's funny, because AI companies are currently spending fortunes on mathematicians, physicists, chemists, software engineers, etc. to create good training data.
Maybe this money would be better spent on creating a Lenat-style ontology, but I guess we'll never know.
We may. LLMs are capable, even arguably at times inventive, but lack the ability to test against ground truth; ontological reasoners can never exceed the implications of the ground truth they're given, but within that scope reason perfectly. These seem like obviously complementary strengths.
> The secretive nature of Cyc has multiple causes. Lenat personally did not release the source code of his PhD project or EURISKO, remained unimpressed with open source, and disliked academia as much as academia disliked him.
One thing that's not mentioned here, but something that I took away from Wolfram's obituary of Lenat (https://writings.stephenwolfram.com/2023/09/remembering-doug...) was that Lenat was very easily distracted ("Could we somehow usefully connect [Wolfram|Alpha and the Wolfram Language] to CYC? ... But when I was at SXSW the next year Doug had something else he wanted to show me. It was a math education game.").
My armchair diagnosis is untreated ADHD. He might have had had discussing the internals of CYC on his todo list since its first prototype, but the draft was never ready.
Lex Fridman interview with Lenat a few years back. https://www.youtube.com/watch?v=3wMKoSRbGVs
It is worth plodding ahead with symbolic AI research because:
- much less resource hungry / planet warming
- auditable chains of inference
Of course there is still the downside that it doesn't work very well.
Neither does the mainstream approach, given the whole hallucination problem.
But then again, humans "hallucinate" in this sense all the time, too.
I thought the end goal is to make something that's way more than human in capabilities.
The thing is, if we produce something that is worse than humans (and right now LLMs are worse than humans with good search indexes at hand), there's not much point doing it. It's provably less expensive to bear, raise, and educate actual humans. And to educate a human, somehow you don't need to dump the whole internet and all pirated content ever created into their heads.
> LLMs are worse than humans with good search indexes at hand
But "good search indexes" are all LLM-based.
:)
"Their topmost distinction was between things with souls and things without souls. And large trees were in the former category, whereas small trees were in the latter category…"
This reminds me deeply of Borges: https://en.wikipedia.org/wiki/Celestial_Emporium_of_Benevole...
To me, that bit of Borges is a reminder that all human taxonomies are limited and provisional. But it seems to me that Cyc and its brethren are built around the notion that a universal taxonomy is important and achievable. I guess it's possible that a useful kind of cognition could happen that way, but it's patently not how people work. If I had gotten to the point where I was forced to define exactly when a tree got a soul, I hope I'd realize that I was barking up the wrong tree.
Lenat in his talks emphasised the infeasibility of building a universal ontology. From what I understand later Cyc modus operandi was a bunch of self consistent micro-theories but not a one huge theory of everything.
Yes, but I'm not sure he fully took in the lesson. Did they really build a bunch of separate domain-specific ontologies? Or, given that it's one system, did they build a universal ontology that's just a big mess?
The lesson I take from the fact that universal ontologies are untenable is that human cognition isn't driven by ontology, so the quest to make a thinking thing out of pile of symbolic logic is one that has no guarantee of succeeding. I think Cyc's whole project is roughly similar to the Frankensteinian notion that if you just put together the right parts and provide a vital spark, you'll get a living being. It might work and it might not, but either way it's not science; it's sympathetic magic with the trapping of science.
Well that's how LLMs got going: a critical mass of context until it ignited from its own gravitational pull.
And yes Cyc ontology wasn't consistent. Lenat's point was that it is impossible to have an ontology consistent. Which makes sense given how there is no consistence across human society or even every individual human.
I wouldn't say that LLMs have "ignited". I think they've gotten steadily better at autocomplete, but I don't think they've crossed any important threshold on the way to the sort of autonomous intelligence that Lenat was after.
I also don't think it was Lenat's original point to have an inconsistent ontology, as evidenced by his early projects. I agree that he eventually had to admit that. What I'm saying is that when he admitted that he should have recognized that it cut at the heart of what he was up to. Something I think borne out by the fact that he spent his whole life on something that didn't succeed on its own terms.
I always had my reservations about the whole AGI thing. And while I'm mightily impressed by Chat-GPT and friends, it's even clearer to me that AGI is not, and will never be, an emergent property of LLM, no matter how large the neural net. And that was likely true for Cyc as well.
I had a particular Cyc success story relayed to me years ago by a customer (not a Cycorp employee), the details of which I cannot divulge, but it was a pretty whopping success and the customer was quite happy with what Cyc had been able to do for them.
So while no AGI, it definitely seems like there was value to be had.
What do you think of the recent anthropic research on how LLMs reason? It is clear from their shallow analysis that LLMs have very serious reasoning weaknesses. I wonder if those weaknesses can be "addressed" if we build LLMs that can do deeper analysis and use RL to self-improve. LLMs improving LLMs would be a very impressive step towards AGI.
I think we are careless in how we use terms. We often say "intelligence" where me mean "sentience". We have studied intelligence for a long time and we have IQ tests that can measure it. The various LLMs (like Chat GPT and Gemini) are scoring pretty well on the IQ tests. So given that, I think we can conclude that they are intelligent, as we can measure it.
But while we have measurements for "intelligence" we don't for "sentience", "agency", "consciousness" or these other things. And I'd argue that there are lots of intelligent life on earth (take crows as an example) that are sentient to a degree that the LLMs are not. My guess is this is because of their "agency" - their drive for survival. The LLMs we have now are clearly smarter than crows and cats but not sentient in the way those animals are. So I think it's safe to say that "sentience" (whatever that is) is not an emergent property of neural net/training data size. If it were, it'd be evident already.
So Gemini/Chat GPT seem to be "intelligence", but in tool form. Very unexpected. Something I would not have believed possible 5 or 10 years ago, but there it is.
As to whether we could create a "sentient" AI, an AGI, I don't see any reason we shouldn't be able to. But it's clear to me that something else is needed, besides intelligence. Maybe it's agency, maybe it's something else (the experience of times passage?). We probably need to ways of measuring and evaluating these other things before we can progress further.
Great post that hits the sweet spot between a dry academic summary and a popular exposition.
Personally I hope that the current wave of AI is over hyped and misunderstood (so that e.g. https://ai-2027.com/ will be a comical footnote one day) and that symbolic reasoning will make a comeback in a new form.
I think there could be a next generation cyc. Current llms have too many mistakes and grounding it with an AI ontology could be really interesting. I wrote more about it here: https://blog.jtoy.net/understanding-cyc-the-ai-database/
I wonder if it's possible to design a new formal language (something like lojban [1], with a grammar strictly based on formal logic) but with better UX so that it can be used by regular folks like me? Maybe combine something like Attempto [2] with a dedicated visual UI?
I would think a language like that could speed up knowledge base construction. Maybe it can also serve as a substitute for natural languages in some situations where we want our communication to be logically airtight.
[1] https://en.wikipedia.org/wiki/Lojban
[2] https://en.wikipedia.org/wiki/Attempto_Controlled_English
On a separate note, I've always wondered how Cyc is pronounced. Is it "sike", or is it "see-why-see"?
I pronounce it like "tsits", because "cyc" in Polish means "tit" (the organ, not the bird), and it feels good.
I've always imagined it was supposed to be pronounced like 'psych', and that it was a corporate bullshit, marketing spelling, of that word.
The etymology is from 'enCYClopedia'.
I see, thanks.
> A heuristic arose which (as part of a daring but ill-advised experiment EURISKO was conducting) said that all machine-synthesized heuristics were terrible and should be eliminated. Luckily, EURISKO chose this very heuristic as one of the first to eliminate, and the problem solved itself.
Hilarious and brilliant.
A very thorough and interesting article -- probably the only thing you ever need read about Cyc. An example of where it shines is the way it carefully pinpoints Lenat contradicting himself on the question of converting English into Cyc's internal form -- Lenat claims that this could likely be done with NNs, but also that NNs are not capable of "common sense", which he himself (correctly) claims is necessary for language translation.
It's not without some flaws, though. The continual harping on how secretive Cyc was doesn't seem fair, given that (with the improbable exception of Meta) none of the currently leading NN-based LLMs have revealed their "source code" (weights) either. Also, I don't agree that Lenat's single-mindedness was a bad thing (I'm glad somebody explored this unpopular concept), nor even that he was necessarily wrong in believing that building a huge, curated fact base is a path to AGI -- the graph showing the exponential increase in the number of assertions in Cyc over time, despite roughly constant headcount, points towards the possibility that a critical threshold was nearby. In fact, the history of NNs is eerily similar: at first found to be promising, then abandoned, then -- against all prevailing wisdom at the time -- found to actually perform very well if you simply scaled them far more than you "should".
OLMo 2 from AI2 is fully open-source, including code, data and recipe: https://allenai.org/olmo
There are others, but OLMo is the most recent and competitive.
Great material!
I always been mesmerized by Cyc (and EURISKO). I've born in late USSR, and my father (then "junior staff scientist" in natural language processing) has several translated books (borrowed from Institute's lab, as I understand mow) which were (again, as I understand now, many years alter) abbreviated translations of proceedings of some "West" conferences about AI. As I read everything about computers, I read these books too, and was deeply impressed by description of EURISKO (it worth noting, that fleet-simulation game was not mentioned, but MOS structure was). I was... 10? 12? years old.
I still think, that this approach much better than modern "multiple huge matrix by huge vector. I understand, that DNNs and LLMs won practically. But I still think that these blackbox pure-feedback based approaches are not so elegant as knowledge engineering... Practical. But in some sense "dirty". Like "dirty hacks" in code.
BTW, part about ABBYY is not exactly correct. I know this because my father became one of the founders and life-long CTO of PROMT, company which is (was? It is not clear after beginning of war, unfortunately) author of successful pre-LLM grammar-based translation systems (Stylus, later PROMT) and partner of ABBYY. ABBYY translation project stated much later than 1990s, because in 1990s ABBYY distributed PROMT technology, and ABBYY itself was (is) experts in OCR and dictionaries (dictionaries is not translation system, of course). Translation project in ABBYY started much later, when their ways with PROMT parted due to several business-related reasons.
> Lenat personally did not release the source code of his PhD project or EURISKO, remained unimpressed with open source, and disliked academia as much as academia disliked him. Most open information concerning Cyc had been deliberately removed circa 2015, at the moment when Cycorp pivoted to commercial applications.
Makes one wonder how much a research being open makes a difference in its real-world success in the current age. Cyc's competitors (LLMs etc.) arguably have a lot to attribute to open public participation for its successes. Perhaps things would have been difference had Lenat been more open with the project?
Probably not, tbh. The issue with Cyc is that it required huge amounts of manual effort to create the rules, while LLMs can learn their own rules from raw data.
There was no machine intelligence in Cyc, just human intelligence.
>while LLMs can learn their own rules from raw data
Supervised vs. unsupervised, but LLMs haven't made any new discoveries on their own ... yet.
[dead]
> There was no machine intelligence in Cyc, just human intelligence.
This can be rephrased for other "AI" projects with exactly the same relevance:
There is no machine intelligence in LLM bots, just human intelligence.
So more of an expert system?
Cyc was exactly an expert system (and those were exactly as central an "AI" technology as LLMs are today, a few rounds of AI hype ago.)
Very much so.
> Lenat personally did not release the source code of his PhD project or EURISKO...<
Now that Lenat is dead, can his PhD project code and EURISKO code be released?
Source code for Eurisko: github.com/seveno4/EURISKO
Source code for AM: https://github.com/white-flame/am
Wonderful essay! OP's tireless research is appreciated.
Yes. Amazing read. This is the kind of content I come to hacker news for.
I think when we look at general intelligence we need to look at multiple methods of reasoning working together. Looking at children there is an instinct to blindly imitate those around them, trying to match behavior without fully understanding it, however the experiences from this behavior then feed into a symbolic understanding in the child’s mind, allowing them to reason about the situation later on
As someone who is mildly skeptical of the current wave of LLM hype and thinks it's hitting a wall, I'm noting a lot of parallels between Cyc and ML/DNN systems' founding stories. We even have tales of reward hacking and other weird meta-shenaniganery. One wonders if some renegade at Cycorp ever trained a CycL to English LLM to fake another GOFAI demo with "real" AI.
As for free lunches, the one free lunch the neural network people got was training data. i.e. being able to scrape the entire WWW and build on that[0]. But that's also a free lunch Cycorp wanted, too! Look at all the various attempts at half-open subsets of Cyc. They were half-hearted attempts to solicit training data from randos, sure, but Cycorp still begged for the free lunch anyway.
But I think the big indictment of AI - both old-fashioned and neural network driven - is that they're chasing this nigh-impossible dream of effectively replicating and replacing a person. I still see the same pattern, in both eras, of toy models working well, narrow problem solvers being effective, but then people trying to scale them up to ludicrous sizes by pumping them full of knowledge in an attempt to achieve generality. Even modern LLMs are only slightly more generalist in the sense that they don't immediately fall over and die when a question is phrased in a way that wasn't expected.
The "bitter lesson" of AI is that the only two things that work are search and learning. LLMs and Cyc both attempted to short-circuit this with massive data dumps - the proverbial "knowledge pump" that needed to be primed. Compression is a good learning substitute, so sure, you can "cheat the test" with big data. But, I mean... how many discoveries were made solely by someone mainlining Wikipedia articles like black tar heroin? At some point, you need a problem space to search, and your learning needs to be done online[1] based on the results of that search. Nobody has cracked that code yet.
Maybe the reason why general AI never took off is that there is no such thing as general knowledge. The biggest source of knowledge the AI community has provided to us is knowledge about how to make more AI.
[0] Whether or not this was legal is a different question.
[1] Specifically, while LLMs think analogously to humans (because that's an easily compressible representation) they do not learn analogously to them.
Open Mind Common Sense is a project now forgotten, a mid 2000s effort to build something like Cyc but in a more open fashion. From the MIT Media Lab, primarily Pushpinder Singh (RIP).
https://en.wikipedia.org/wiki/Open_Mind_Common_Sense
"The long slow failure of Lenat’s project is a strong indictment against the symbolic-logical approach to AI."
That does not follow at all ... this statement comes just after an enumeration of Lenat's attitudinal problems that plausibly played a significant role in its failure.
Are we sure that if we invested in the last ten years the same resources we invested in neural networks (research, hardware, money) the results wouldn't pay off?
What I want to ask is if we know for sure symbolic AI is a dead end and it doesn't worth to pursue that field.
Given the secrecy surrounding Cyc we don't know how close or not they got to something AGI-like. Whatever conclusion you draw will be from very little data.
From a practical standpoint Cycorp seems to have pivoted at least 10 years ago into providing knowledge and inference systems for clients. If customers are willing to pay them then perhaps we can conclude there is value in the technology. But again I'm not aware of any public examples to look at.
> From a practical standpoint Cycorp seems to have pivoted at least 10 years ago into providing knowledge and inference systems for clients. If customers are willing to pay them then perhaps we can conclude there is value in the technology.
This is exactly the kind of thing that suggests they were nowhere close to AGI. Unless you think Palantir has somehow developed AGI too and kept it secret from everyone else.
I have nothing to say about Cyc (apart to that the bitter lesson is bitter indeed, but was never debunked so far; and, also, I hate LLMs). But this line (from accompanying github) deserves some attention, IMO:
> Due to the lack of long-term stability of IA, I have taken the liberty to scrape those from IA and kept them here for safe keeping.
I mean, he is not wrong, and when I want to preserve something code-related, Github comes to mind as a safe place... which is crazy, if you think about it for a moment. And the fact that we are starting to use Github as a backup for IA is almost comically grotesque. I don't have a solution, but it's worth reminding that we sure have a problem.
It's worse. GitHub is owned by Microsoft and exists as a community resource only as long as Microsoft's benevolence holds out.
My own thought process is that it's like Usenet or arXiv: The collective value of Github is so high that even if Microsoft loses interest, someone else will buy it from them or clone the content elsewhere. That said, I do wish there was an alternative that didn't involve a for-profit corporation.
What does it mean to you to hate LLMs?
You sound like ELIZA.
Have you ever not hated LLMs?
What did you mean when you said "Have you ever not hated LLMs"?
I started loving LLMs when I cussed at them enough that they started cussing back.
Yes? I don't hate LLMs and I don't think I ever have. They are super useful and neat, I use them all the time from smolguys on my computer to Claude in the cloud. It's like saying "I hate forks" or something, like why do you hate a generic tool that has many applications? I love that I have these things for free at home forever, my personal wiki and time tracker have both gotten way more useful ever since I baked an LLM in.
I think they're imitating Eliza.
Please let there be an eccentric billionaire who has backed up the internet archive for when it goes bust.
The game mentioned in this article reminds me of the story about heuristics told by Richard Feynman in a lecture: https://youtu.be/EKWGGDXe5MA?si=z9TdlWflOkY6b9Qk&t=4249
I was wondering then who was the guy he talked about.It is great that this article provides so much details of it.
For me, the futility of GOFAI is obvious from simple introspection: is my own brain capable of chaining through vast networks of propositional statements? I wish! I can barely hold a 5 node decision tree in my head while playing a board game. At the same time, my brain is amazingly good at parsing visual and linguistic stimuli.
Still, you have to respect the man's dedication to his quixotic quest.
But wouldn't that kind of reasoning also say that helicopters are impossible because no flying animal works that way?
For a more reasoning-adjacent example, "conventional" chess AIs don't really work like brains do, either, but they're pretty good at playing chess.
Yeah but helicopters fly. The problem with cyc is that it never really got off the ground.
Symbolic AI wasn't a failure but it never really worked for any problem space without rigid rules you could define and that's with some of the greatest minds trying for decades to make it happen.
At this point real-world intelligent symbolic AI only exists in fiction and our collective imaginations. Technically, no one has proven it can't work, but at some point you have to face the reality.
This is more interesting, and if I remember correctly, they had some interesting ideas:
Two AI Pioneers. Two Bizarre Suicides. What Really Happened?
https://www.wired.com/2008/01/ff-aimystery/
Can't we combine symbolic AI with neural networks into something that exceeds the capacities of both?
“Obituary” is a poor word choice. A shame, because the article otherwise is detailed and interesting.
It would be cool to try to generate the "knowledge" as in Cyc automatically, from LLMs.
Or vice versa - perhaps some subset of the "thought chains" of Cyc's inference system could be useful training data for LLMs.
When I first learned about LLMs, what came to mind is some sort of "meeting of the minds" with Cyc. 'Twas not to be, apparently.
I view Cyc's role there as a RAG for common sense reasoning. It might prevent models from advising glue on pizza.
You must not read ingredients on products or eat library paste. ;)
Also: MEAT GLUE (transglutanimase)!
https://en.wikipedia.org/wiki/Transglutaminase
https://en.wikipedia.org/wiki/Molecular_gastronomy
https://lyndeymilan.blogspot.com/2010/01/question-what-is-oc...
https://www.youtube.com/watch?v=oA8sNJjlQzU
https://www.youtube.com/watch?v=XAVO6hzNTns
sure but the bigger models don’t make these trivial mistakes, and I’m not sure if translating the LLM english sentences into LISP and trying to check them is going to be more accurate than just training the models better
The bigger models avoid those mistakes by being, well, bigger. Offloading to a structured knowledgebase would achieve the same without the model needing to be bigger. Indeed, the model could be a lot smaller (and a lot less resource-intensive) if it only needed to worry about converting $LANGUAGE queries to Lisp queries and converting Lisp results back into $LANGUAGE results (where $LANGUAGE is the user's natural language, whatever that might be), rather than having to store some approximation of that knowledgebase within itself on top of understanding $LANGUAGE and understanding whatever ad-hoc query/result language it's unconsciously invented for itself.
Beyond just checking for mistakes, it would be interesting to see if Cyc has concepts that the LLMs don't or vice versa. Can we determine this by examining the models' internals?
I first heard of it in a 1990s book saying it showed a photo of running cars when the users queried it for "people sitting".
One thing that LLMs seem to be missing, even after ingesting the whole web, is the super-basic knowledge that every human learns before the age of 3. Most of that stuff in not written anywhere, because it's totally obvious to anyone who can read. A knowledge base like Cyc can be useful to fill these gaps, because it was made for an AI with zero base knowledge and explicitly specifies even the most obvious concepts.
like what, for example?
"As of 2025, after more than 80 years of research on Artificial Neural Networks, there remains no evidence of their general intelligence. Even after priming the pump with nearly every book ever written, all available scientific articles, and all publicly available text on the internet (and consuming more power and resources than entire countries), they routinely fail at basic tasks and generate nonsensical output and 'slop' that has already degraded the internet as a source of training data for future systems. The long slow failure of ANNs is a strong indictment against the statistical/neuromorphic approach to AI."
Exactly.
On top of it, Cyc's whole 40-year run expense of "200 million dollars, 2000 person-years" is... What LLM companies waste each month, for slightly less hallucinations and getting the right number of fingers most of the time?
LLMs are clearly useful, even if flawed; conversely, the author had to struggle to find uses of Cyc. Additionally, neural networks were already showing promise before the scale-ups; AlexNet was trained on consumer hardware and outperformed all other solutions to date.
I see your point, and I agree that the way that paragraph was initially posed was perhaps more dismissive than it should have been... but to disprove it, you need proof that Cyc was a working approach for anything, and said proof appears to be lacking.
There's plenty evidence of the general intelligence of LLMs. And a site that has climbed to the 6th most visited in 2 years is clearly not failing to be generally useful.
I do think the author was more scathing than necessary here though
I enjoyed this read and agree Lenat was a grifter, which is easy to see based on contracts and closed source. But I dislike how the article seems tilted towards a hit piece against search, heuristics, reasoning, symbolic approaches in AI, and even striving for explainable/understandable systems. It's a subtext throughout, so perhaps I'm misinterpreting it.. but the neats vs the scruffies thing is just not really productive, and there seems to be no real reason for the "either/or" mentality.
To put some of this into starker contrast.. 40 years, 200 million dollars, and broken promises is the cost burned on something besides ML? Wait isn't the current approach burning that kind of cash in a weekend, and aren't we proudly backdating deep-learning to ~1960 every time someone calls it "new"? Is a huge volume of inscrutable weights, with unknown sources, generated at huge costs, really "better" than closed-source in terms of transparency? Are we not very busy building agents and critics very much like Minky's society of mind while we shake our heads and say he was wrong?
This write-up also appears to me as if it were kind of punching down. A "hostile assessment" in an "obituary" is certainly easy in hindsight, especially if business is booming in your (currently very popular) neighborhood. If you didn't want to punch down, if you really want to go on record as saying logic/search are completely dead-ended and AGI won't ever touch the stuff.. it would probably look more like critiquing symbolica.ai, saying that nothing like scallop-lang / pyreason will ever find any use-cases, etc.
> and agree Lenat was a grifter
I worked at Cycorp for 3.5 years, so I can say:
While I haven't been surprised to see Cyc become more and more clearly a failure, Doug Lenat was no grifter. To the very end, he (alongside a handful of others at the company) was the truest true-believer I've ever known. Cyc was his life's mission, and he never doubted that.
What do you think made Lenat work so secretively? I suspect his ideas could have advanced a lot quicker if he'd been plugged into the community, sharing and getting feedback. In research it's almost never a good move to pursue a dream in isolation.
I'd say that being a believer doesn't necessarily conflict with being a grifter. In fact the true believer is practically obligated to engage in tricks, because the ends justify the means. For example taking government contracts for anti terrorism because the problem isn't well defined, your results are hard to check, and the money is going to be good if temporary. Anything where you don't really expect to provide promised value is always worth it because you're going to "pay it back" with big results later. For more recent stuff along the same lines, colonizing mars and getting away from oil are like that too, because for the true believer stuff like this is worth a little stock manipulation or whatever else is required.
Terminology aside, you're making a lot of accusations/character portraits that I don't think you have any basis for.
Doug was an uncynical person.
Doug Lenat was a gem, and I am sad he is no longer with us. He had a lot of interesting ideas for building a large symbolic AI system.
Just because Cyc - and symbolic AI in general - failed to scale like DNNs (including LLMs) doesn't mean that its ideas were all bad.
Grifting suggests a level of cynicism, that one knows that one is selling snake oil.
I don’t know Lenat and don’t have an opinion one way or another. But be careful suggesting someone is grifting verses just believing in an idea that ultimately doesn’t come to fruition.
> even striving for explainable/understandable systems
It's been almost 6-8000 years since the advent of writing and we still cannot explain or understand human intelligence and yet we expect to be able to understand a machine that is close to or surpasses human intelligence? Isn't the premise fundamentally flawed?
I think I'd remain interested in more conclusive proof one way or the other, since by your logic everything that's currently unknown is unknowable.
Regardless of whether the project of explainable / understandable succeeds though, everyone should agree it's a worthy goal. Unless you like the idea of stock-markets, resource planning for cities and whole societies under the control of technology that's literally indistinguishable from oracles speaking to a whispering wind. I'd prefer someone else is able to hear/understand/check their math or their arguments. Speaking of 6-8000 years since something happened, oracles and mystical crap like that should be forgotten relics of a bygone era rather than an explicit goal for the future
It is actually incredibly silly to expect full explain ability as a goal because any system sufficiently intelligent to do basic arithmetic will have behavior that is inexplicable.
Why call Lenat a "grifter"? How's that about punching down, not to say, pissing on dead peoples' graves?
Very disappointing as otherwise your comment is insightful and on point.
I like the guys general ideas about research but um. Did you see the section describing contracts? Article states 50% of funding came from the military. People would be freaking out if they heard the same about Google, Facebook, or OpenAI.. for good reason.
I'm not a fan of weaponizing AI, and I think that's what we're talking about. Either it was a glorified CMS, in which case the presentation as AI was dishonest and cynical. Or it really was AI, in which case it was weaponized research.
If we're talking about graves, then it might be good to also consider all of the ones that you're not mentioning, the ones presumably resulting from the details about where the money came from. How many? How many of those deserved it and how many were bad inferences? I guess we'll never know.
Oh, OK. Well I don't call that a grifter, just an ordinary, garden variety, techie. Many (not all) do that; actively seek funding from militaries for their work.
E.g., just this week MS fired two people for protesting the use of Azure to power the Palestinian Genocide [1].
When people talk about the military-industrial complex, what they really should be talking about is the military-FAANG complex. AI and military intelligence are both the same sad joke.
Lenat was no different in that, so I don't think it's fair to call him a grifter. I do think it's fair to call him out on being an asshole who put money above peoples' lives.
Btw, I've released some of my free stuff under a modified GNU 3.0 with an added clause that prohibits its use for military applications. I've been told that makes it "non-free" and it seems that's a bad thing. Lenat is only one nerd in a long line of nerds that need to think very hard about the ethics of their work.
____________
[1] https://apnews.com/article/microsoft-protest-employees-fired...
They were protesting about this:
https://www.972mag.com/microsoft-azure-openai-israeli-army-c...
This is the best overview I have ever seen, and I have had a passing interest in Cyc since I first read about EURISKO. Hands down, this is great: thorough, historical ... perhaps a little more negative than I would like.
My personal "take," which is worth nothing, is that something like Cyc is necessary but not sufficient for a serious AGI. Many other components will probably be required.
Yes, we are gonna need natural language parsing, and a good one, unless we can make the whole world speak in lojban. It would certainly speed up the collection of assertions.
I suspect embodiment is also a requisite, wherein one can acquire experiences and test premises. The ability to explore your surroundings, to move, to investigate. To jam your sensory cluster where it does not belong. To drop a block and see it fall. To drop a balloon and see it rise -- surely that must suggest an avenue of investigation.
I would be surprised if neural networks will not make a showing. Recognition through the fuzziness and quirks of reality is needed before you can sufficiently abstract into various rules. That isn't a new species of animal, it's a cat without fur and with an unusual number of limbs (Sphynx with a missing leg). Recognize, then expand the knowledge: cats usually have fur, but some breeds do not. Cats usually have four limbs, but they can be lost. But before you can say that, you still have to recognize the catness of the beast before you.
Another probable component: emotion. e-motion. Motion. Drive. In a way, hunger is an emotion, as is a full bladder. These are prompts to do something that one might return to equilibrium and focus on the higher level problems. Buildup of carbon dioxide, that one surely prompts you. These are currently tied to our embodiment, but parallels can be worked out, such as Opportunity's last ... words ... "My battery is low and it's getting dark." Right now, our attempts at intelligences are still reactive. Waiting for a prompt, waiting for a problem to be put before them. I suspect a strange loop (or really a series of interconnected ones) might be one of the final components, with the emotions serving as a way to keep it from lapsing into catatonia. Boredom would be one. Most of our human emotions would be counter-productive, but probably not all, and there may be analogues.
As a subcomponent of that, I believe in laughter. Babies laugh a lot. We laugh at jokes. No, we laugh at new jokes; old jokes grow progressively less and less funny. Laughter seems to be often prompted by a new connection, and it is both a reward and a communication.
I have covered both textual ingestion and experiential evidence, but let's throw in another one: I expect that the ability to examine an expert system and replicate it, to incorporate it into its mind, is one of them. Think about those old flowcharts in hardware books which enable you to diagnose a computer with boot issues. We can do it and I expect an AGI ought to be able to examine other rulesets for ingestion as well.
In short, I am suggesting that our first AGI may require all of the things a human does to be intelligent. We may find that we can do without one or two components. While I most certainly do not want a Human in a Box to be the end product, for a number of reasons, it does seem reasonable to me that we ought to incorporate the different things we already know about the human mind into something we are hoping to be an equivalent.
However, I strongly suspect that giving it full access to something like Cyc would beat just chucking encyclopedias at it and telling it to read.
Lenat beat Musk at his own game. Musk has only been promising "full self driving next year" for 10 years.
Doug Lenat managed to make similar hopeless promises for nearly a half century.
I think Cyc should worry that I can't distinguish whether real or April Fools joke ...
This all brought up some interesting memories from my time funding Cyc.
Yep, they completely missed the boat. They tried to use concepts without actually modeling concepts, making a huge mess of contradicting statements which actually didn't model the world. Using a word in a statement does not a concept make!
>> As of 2025, 9 years after the knowledge pump had been primed, there is still no sign that Cyc would ever achieve general intelligence. The long slow failure of Lenat’s project is a strong indictment against the symbolic-logical approach to AI.
Really? CYC, a strong indictment against SAT solving, though unrelated to it? A strong indictment against automated theorem proving, though unrelated to it? A strong indictment against planning and scheduling, though unrelated to it? A strong indictment against program verification and model checking, though unrelated to it? A strong indictment against Knowledge Representation and Reasoning, though unrelated to it?
Or are those "symbolic-logical approaches to AI"? Some of those fields use heuristic search, but then so does Neural Networks research. What's gradient optimisation? It's a search algorithm that follows the gradient of a function based on the strong assumption that the function has its optimum at the location of the searched-for datum. So, a search algorithm; with a heuristic cost function.
How about probabilistic AI? Is that "symbolic-logical AI"? I mean the Bayesian calculus is a form of logic after all, and probabilities are symbolic right up to the time when they turn into statistics when you have to give them values [1].
How about neural networks again? Everyone will tell you they are "sub-symbolic" but in truth their "continuous" weights are finite-precision floating point numbers, so they're basically symbols, with a very large vocabulary. And in any case Neural Nets, like everything else, are running on a computer, a symbol manipulation device based on Boolean algebra and logic gates. Is that symbolic, logical, or neither?
See what happens when you make up new terminology as you go along and then pretend like it's some kind of agreed-upon definition that everyone agrees upon? Yes I know. My first language is not English either, but the author has clearly gone and er delved into old papers about CYC so I don't see that they have any excuse to make all those unsubstantiated, and unsupported, assumptions about what "symbolic-logical AI" is.
And as for CYC and Lenat: it's easy to disparage a man's work when he's dead and can't defend it. It's easier still if you're just repeating the same talking points that has circulated in the grapevine for who knows how long: blah blah CYC failed, blah blah symbolic AI failed blah. The fact of the matter is that CYC is closed source, very few people can use it, and nobody, outside of Cycorp, knows its true capabilities. A bit like "OpenAI"'s latest "reasoning" models, all we can do is speculate and pretend we know what we're talking about, when we clearly don't. I mean they. The author.
_________________
[1] Which is what you have to do if you want to use probabilities in the real world. And if you feed the Mogwai after midnight they turn into Gremlins.
First, apologies for writing "symbolic-logical AI". It should have been "symbolic-logical AGI".
Second, I agree that the failure of the symbolic-logical approach to AGI is not justified in-text. I believe that it will only be a small component to the first AGI system, as an API for the AGI to use some tools like a calculator, a proof verifier, etc. A way to invoke the "crystalized intelligence" as it is. Nevertheless, since it is not justified in-text, and a justification would probably as a short essay, and is not necessary for the essay itself, it has been removed.
I'm sorry you felt the need to edit your text, it was not my intention to cause you to do that.
They are specifically talking about "general intelligence" here, something which hasn't been achieved and so we can probably classify all attempts towards it as failures at this point.
I don't think that the author meant to write off the entire field of symbolic reasoning because Cyc didn't achieve general intelligence but I think it's fair to say that the lack of progress Cyc made towards it does make it less likely that purely symbolic reasoning will lead us to AGI. That's not to say it might not play a part in AGI or that it doesn't have other uses.
[dead]
[dead]
[flagged]
[flagged]
https://tvtropes.org/pmwiki/pmwiki.php/Main/InMyLanguageThat...
why are you cursing people to a 3-4 hours trip to tv tropes :(
Just resurfacing now.
I love that tvtropes has the same effect on me that snopes did in 1998
"Cyc has failed to reach intellectual maturity, and may never will."
'may never will'?