ChatGPT has garnered a tremendous reaction since being released just over a week ago. The output it has generated is truly remarkable and has many wondering what impact AI will have on knowledge work.
Some are terrified that it will take their jobs, and others have dismissed it as a “nice toy”.
Let’s take a look at how these AI models work to understand their strengths and weaknesses, in order to understand the potential impact of AI on knowledge work and the accounting industry in particular.
How AI’s work
At the most fundamental level, modern AI’s like ChatGPT generate content using statistical methods. ChatGPT specifically has two components, a “generator model” and a “scoring model”. The generator model creates output and the scoring model rates the quality of the output. The scoring model provides feedback to the generator model so that it improves over time. Both of these models were trained by humans on what “good responses” look like. Based on a prompt, ChatGPT generates responses that are likely to be rated as “good responses” to that prompt, but ChatGPT does not understand the prompt or even what it is generating.
How can you generate valid information without understanding?
Let’s take a look at a simple example to understand how meaningful data can be generated without fundamental understanding of the inputs or the outputs. We’ll start with a list of “words”. Some are valid, others are not. We’ll then try to generate new “words” based on what we know from our original list. Then we’ll evaluate the results. Here’s our starting list of three letter “words”. The first 4 are valid and the last 2 are not (as you can tell by the red underline):
All of the valid words end with the “m” looking symbol and half of them start with the “peace sign”. Let’s use this information to generate other “words” that match this pattern:
People fluent in Wingdings who read this output are amazed! Based on a limited set of inputs, we were able to generate real words! Here they are translated to English:
Ate
Ape
Awe
Are
Axe
Ale
Aye
You can put these words into Word and change the font to Wingdings to verify for yourself.
The amazement of our Wingdings readers doesn't change the fact that the content we generated is completely meaningless to us. Without “translating” the words to English, we can’t tell which are verbs or nouns or even if they mean anything at all. It’s the same with the content generated by AI, albeit on a much grander and more impressive scale.
Ultimately, AI has no “fundamental understanding” of the content it’s generating, which means that logical fallacies and irrelevant information are likely to be included in its output.
What does ChatGPT have to say about this?
Take a look at this response from ChatGPT when asked if AI could replace knowledge work:
It's unlikely that AI will completely take over knowledge work. While AI is capable of completing many tasks, there are certain aspects of knowledge work that require creativity and critical thinking, which are difficult for AI to replicate. Additionally, many people enjoy the social aspect of knowledge work and may not want to be replaced by machines. That said, it's possible that AI could assist with some aspects of knowledge work, such as data analysis and research, which could make some jobs more efficient.
This is an impressive response, and I actually agree with the conclusion, but notice the third sentence: it is especially instructive in its irrelevance. Given how the economy works, whether or not people enjoy working does not matter. Just ask the millions of factory workers who were replaced by robots over the last 50 years. This sentence makes it clear that there is no “understanding” of the dynamics involved in AI replacing knowledge work in human civilization.
Instead of truly understanding the question presented, the response follows a basic pattern that is is generally relevant to questions about something being replaced:
Is X better than Y in all regards?
Do some people like having Y over not having Y?
Can X and Y coexist?
While this is a helpful pattern to follow with this type of question, it can’t be strictly applied in this case, as the second question is not relevant in this scenario.
Approximate vs Actual Intelligence
Humans create content in a completely different way than AI. We create new things based on fundamental understanding of the concepts involved. This is often referred to as “reasoning from first principles”. We are not just recognizing patterns. We know the why and the how. True knowledge is not created by copying the characteristics of other knowledge.
Just like Newton’s laws are an approximation of the laws of the universe under certain conditions, current AI is an approximation of intelligence, not actual intelligence. That doesn’t mean Newton’s laws are unimportant or irrelevant. In fact, they were a tremendous step forward for humanity and were used to accomplish incredible things, like putting people on the moon. I believe the same is true of AI for knowledge work today – it is a tool that can be used to propel us to new heights by augmenting and assisting knowledge workers, not by replacing them.
Approximate intelligence is good enough in so many cases in the modern economy:
- Formatting documents and data
- Data extraction and data entry
- Identifying patterns in large datasets
- Anomaly detection
- Idea gathering
- Research and data summarization
But Actual Intelligence is needed as well:
- Understanding content of documents and implications of data
- Validating the reasonableness of ideas and accuracy of conclusions
- Understanding underlying root causes of outcomes
How AI could transform audits
We’ve established that AI does not “understand” anything, but that doesn’t mean it can’t transform the way audits are performed. Here are a few ways AI could augment audit teams to improve quality and reduce the time spent by both the auditor and the client:
Generating ideas of risks particular to your client and their industry
Identifying unusual transactions and other anomalies in large datasets
Generating documentation of the audit procedures performed.
Rejecting evidence provided by the client if it doesn’t match what was requested
Mining documents and recordings for key information related to changes that have happened within the company
Extracting and transforming data from documents into testing tables for audit documentation.
Assessing whether evidence meets certain characteristics, e.g. “Were any journal entries created and posted by the same individual?”
And here are a few more suggestions straight from ChatGPT:
Automating the collection and analysis of data to identify trends or anomalies.
Developing predictive models to identify areas of risk or areas that require more attention.
Developing algorithms to identify potential fraud or other financial irregularities.
Improving the accuracy of financial forecasts and projections.
Streamlining the audit process by automating routine tasks.
Providing real-time feedback and insights to auditors during the audit process.
Enabling auditors to access and analyze large amounts of data quickly and efficiently.
Enhancing auditors' ability to identify and prioritize high-risk areas.
Providing auditors with instant access to relevant information and expertise.
Helping auditors to comply with complex regulations and standards.
What AI can’t do in an audit
Ultimately, AI doesn't have understanding, and therefore, cannot reliably apply judgment to a particular scenario. This is fundamental to all audits and so the ultimate outcome of the audit, the auditor’s opinion, is not in danger of being replaced with AI.
Another example comes from writing code. Stack Overflow banned responses from ChatGPT because they were consistently wrong. The answers had the semblance of accuracy, but in software, as with many knowledge-based tasks, identifying the correct answer often requires understanding of the fundamental concepts from first principles, not simply imitating other correct answers.
Conclusion
As a former auditor turned technologist, I’m not worried about AI taking over either field. In fact, I think we will see a renaissance of knowledge work, where humans are freed from the mundane, time consuming tasks that don't require Actual Intelligence, where an approximation is more than sufficient, or provides a solid starting point. And this perspective informs our goal with UpLink: to make work more enjoyable and fulfilling for millions of people.
TL;DR:
AI doesn’t “understand” the prompts it receives or the output it generates
AI can perform many tasks that imitate intelligence
Many tasks performed by knowledge workers can be performed by AI, and
Many cannot.