핵심 요약
골드만삭스 출신 분석가가 전하는 AI 활용 핵심 원칙 중 첫 번째 단계인 검증(Verify)에 관한 내용이다. AI가 그럴듯하게 내놓은 결과물과 실제 정답은 엄연히 다르므로, 프로세스 자체에 검증 단계를 구축하는 것이 필수적이다. 모델의 최종 답변이 틀리더라도 추론 과정을 분해해 활용할 수 있으며, 결국 AI 시스템의 성능은 기반 데이터의 품질에 달려 있다.
주요 포인트
- **결과물 검증(Verify)의 필수성**: AI는 틀린 답변도 매우 완성도 높고 그럴듯하게 출력하므로 반드시 사실 여부를 재확인해야 한다.
- **추론과 최종 출력의 분리**: 골드만삭스 CIO에 따르면, 모델의 최종 결론이 틀리더라도 문제를 분해하고 접근하는 추론 과정 자체에서 가치를 얻을 수 있다.
- **데이터 품질이 핵심(Garbage In, Garbage Out)**: 정제되지 않은 지저분한 데이터를 AI에 넣으면 마법처럼 해결되는 것이 아니라 잘못된 답변만 더 빠르게 도출될 뿐이다.
- **프로세스 내 검증 단계 내장**: AI에게 최종 답변 전 수치와 사실을 재점검하고 출처를 표기하도록 프롬프트나 파이프라인 수준에서 검증을 강제해야 한다.
So, I'm going to give you everything that I learned after working at Goldman Sachs, one of the biggest financial firms on the planet. These are principles I still follow to this day in everything that I do with AI. A company like this doesn't have the luxury of making mistakes, and one wrong data entry or hallucination can cost them millions of dollars, or even worse, their reputation. So, if one of the biggest and most trusted financial firms on the planet follows these principles, it's because they work. And even if you're not working at a big firm, these principles will completely change how you use AI and the results that you get out of AI. So today I'm going to share the five AI principles I learned from Goldman Sachs which I call vault and show you how to apply them to your AI projects. So let's get into it. Okay, so V is to verify the output. And just a reminder, this is the framework I created from the principles I learned while working there. And these principles apply whether you're using claude code, codecs, or whatever new tool comes out next month. So Verify, I think a lot of us, myself included, can get a little too comfortable with AI because it gives you something that looks finished and it will deliver it to you super confidently. It's formatted perfectly and it probably took about 30 seconds. But something that looks finished and something that's actually correct are obviously two very different things. So Marco Agenti, Goldman's CIO, has made this distinction between the reasoning of a model and its final output. Basically, the way a model breaks down and analyzes a problem can still be useful even when the final answer it gives you is wrong. So you can use the breakdown to get value from it, but you still need to verify the result. And this actually starts before the AI even generates an answer. So my role at Goldman was in business intelligence. And every business intelligence analyst in my role had to pass something called data school. We had to understand how data gets pulled in, how it gets cleaned, how it gets updated, and how you turn all of that into something that people can actually use to make a decision because every report, every dashboard, every automation, and now every AI system is only as good as the data that's actually powering it. And your company's data is probably one of the biggest advantages that it has because nobody else has that exact information. They don't have your customers, your sales history, your internal processes, your support tickets, all of that kind of stuff. It's gold. But if that data is outdated or duplicated or messy or just wrong, then plugging it into AI doesn't magically fix it. It just gives you the wrong answer faster. So when I use AI tools, I like to build verification directly into the processes. You could give the model a prompt like, "Hey, you know, before you give me the final answer, recheck every number and factual claim. Cite the source for each one and flag anything that you're not fully 100% confident about." And you don't have to manually verify every single word that it gives you. If you're creating a report, maybe there are only two or three numbers that could actually change the decision. So have the model site those numbers and then you can spot check them yourself. And obviously for higher stakes projects you can even build a separate AI reviewer or multiple you know like a team of AI agents that are just there to review. They'll check the first output before you see anything and then the whole system can iterate once again before you see anything. And that's an additional check. But important claims still need original sources, deterministic tests or human review. So verify the data going in, verify the important outputs coming out and build review into [music] the process. But once you can trust the inputs and the outputs, you still have to decide whether AI should be in the workflow at all. So moving on to a augment, which means augment don't replace. Now my full-time job at Goldman was basically building automations, reports, dashboards, and systems that made teams more efficient. [music] And across the teams I worked with, there were people doing some version of that work. None of this was some brand new AI initiative. You know, Goldman had been building automated data and risk systems for decades. and teams across the firm were doing this kind of work before the current wave of AI agents hit the market. Which means there were already a crazy amount of problems worth solving before anyone started talking about AI agents. There were manual reports being built that took hours. You know, data had to be moved between systems. Teams were waiting on information from other people. People would enter something incorrectly with a fat finger or skip a step or or two people would follow the same process in two slightly different ways. [music] Because at the end of the day, humans are inconsistent. We get tired. We miss steps. comm