<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://www.artkreimer.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.artkreimer.com/" rel="alternate" type="text/html" /><updated>2026-06-07T20:29:43+00:00</updated><id>https://www.artkreimer.com/feed.xml</id><title type="html">Art Kreimer</title><subtitle>Product Leader writing about Artificial Intelligence, Machine Learning, Product Management, and Leadership.</subtitle><author><name>Art Kreimer</name></author><entry><title type="html">The Role of a Product Manager in the AI Era</title><link href="https://www.artkreimer.com/role_of_pm_in_ai_era/" rel="alternate" type="text/html" title="The Role of a Product Manager in the AI Era" /><published>2026-04-26T00:00:00+00:00</published><updated>2026-04-26T00:00:00+00:00</updated><id>https://www.artkreimer.com/role-of-pm-in-ai-era</id><content type="html" xml:base="https://www.artkreimer.com/role_of_pm_in_ai_era/"><![CDATA[<p>In 2024, I wrote a piece <a href="/6_ways_pm_role_will_change/">6 ways the role of a product manager is going to change in the next decade</a> for Scotia Digital’s Future Fest Craft Magazine with six predictions about how the PM role would change over the next decade. Two years later, most of them have already happened. That tells you something — I was too conservative on the timeline. Predicting the next ten years now feels like a category error. The window I’d actually defend is twelve to twenty-four months, and even that might be generous.</p>

<p>Looking back at that piece, plus my <a href="/The-Role-of-a-Product-Manager/">2022 post on the role of a PM</a>, I got several things right. But the most important shift I missed wasn’t a skill or a tool. It was a shift in where the constraint sits.</p>

<h2 id="the-bottleneck-moved">The bottleneck moved</h2>

<p>The traditional 6:1 ratio of engineers to PMs assumed engineering was the expensive, scarce thing. Six people building, one person figuring out what to build. That assumption has cracked.</p>

<p>Andrew Ng claims his teams now build in a weekend what used to take six engineers three months.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> I’d take the specific numbers with salt — it’s a hot take, not data — but the directional point is hard to argue with. The constraint has moved. The question is no longer <em>can we build this?</em> It’s <em>what exactly should we build, and should we build it at all?</em></p>

<p>Some AI-native teams have flipped the ratio entirely, running closer to two PMs per engineer. I’m skeptical of that as a stable equilibrium. What I think is actually happening is closer to what Cat Wu describes: the roles aren’t shifting in headcount, they’re collapsing into the same person.<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> An engineer with strong product sense can move without a PM. A PM with strong building skills can ship code. The interesting question isn’t the ratio — it’s whether enough product judgment exists in the room.</p>

<p>So when I say “product management has become the bottleneck,” I don’t mean the job title. I mean the function: deciding what’s worth building when building is no longer the limiting factor. That function used to be split across PM, design, and engineering through a series of handoffs. The handoffs were the slow part. Now the slow part is the decision itself.</p>

<p>This is the lens I’d hold onto for the rest of the piece: every shift below is downstream of this one.</p>

<h2 id="what-my-2024-predictions-got-right--and-where-they-were-too-small">What my 2024 predictions got right — and where they were too small</h2>

<p><strong>What was directionally right.</strong> AI-assisted PM work has come true faster than I described — synthesizing research, drafting PRDs, writing competitive analyses in minutes is the present baseline, not a future state. The push for AI fluency has accelerated past my framing of “baseline ML concepts.” Today’s bar is evaluation design, model failure-mode awareness, and direct work with agentic systems. And the focus on customer-centricity holds. When anyone can ship a product in a weekend, deep user empathy and problem understanding is one of the few remaining moats.</p>

<p><strong>What was too small.</strong> I predicted “the rise of the product visionary” - PMs who anticipate trends and translate them into compelling roadmaps. Not wrong, but too abstract. Visionaries operating on six-to-twelve-month cycles are too slow for what’s emerging. The actual skill is more operational: <strong>product taste</strong>. Knowing what’s good before users tell you, evaluating fast, deciding faster.</p>

<p>I also under-described how much the role boundaries would dissolve. I called out cross-functional collaboration; what’s actually happening is structural. LinkedIn replaced its Associate PM program with a “Product Builder” program that trains people across product, design, <em>and</em> engineering simultaneously. That’s not adjacent to the role. It’s a redefinition of who gets hired into it.</p>

<p>And I missed the bottleneck shift entirely.</p>

<h2 id="whats-getting-disrupted">What’s getting disrupted</h2>

<p><strong>The information-mover PM is in trouble.</strong> Nikhyl Singhal’s framework of “information movers vs. builders” is the one I keep returning to.<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> The information mover synthesizes research into a doc, moves slides between teams, aligns stakeholders, writes the PRD. Useful work, but increasingly automatable. AI can do the information movement — not as well as a skilled PM, but well enough, fast enough, that synthesis-as-edge is gone.</p>

<p>The uncomfortable implication: a meaningful share of “successful” PM careers were built on coordination skills that aren’t differentiating anymore. I’ve watched this play out in my own org - the PMs who struggled when AI entered their workflow were the ones whose primary value was moving information cleanly between teams. The ones who thrived had a point of view about what to build and could defend it.</p>

<p><strong>The iteration cycle has collapsed.</strong> Cat Wu puts the compression in one sentence: “What used to take six months now can take one day.” That doesn’t just change how teams operate — it breaks the premise of annual roadmapping and the PM as long-horizon-thinker. Jenny Wen describes the same compression on the design side: “Vision used to be six months out. Now it’s three to six weeks.”<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup></p>

<p>The 12 months roadmap is quietly becoming a liability. It implies a level of certainty about what will matter in more than three months that nothing about the current environment supports. Some teams started to replace roadmaps with operating principles — clear enough to guide daily decisions, loose enough to update weekly.</p>

<h2 id="what-the-role-is-becoming">What the role is becoming</h2>

<h3 id="the-pm-as-judgment-engine">The PM as judgment engine</h3>

<p>The scarce resource in product development is no longer execution speed. AI provides that. It’s the quality and speed of decisions made under uncertainty.</p>

<p>Andrew Ng says his teams are “increasingly relying on gut” for decisions that used to require weeks of A/B testing — he calls A/B testing one of the slowest strategies in his portfolio. When you can prototype in a day, waiting a week for statistical significance is a competitive disadvantage, not a discipline.</p>

<p>This is what the role is converging on: someone who can make high-quality decisions quickly, on incomplete data, and be right often enough to matter. It’s the part of the job that was always the point but was usually buried under coordination work. Now the coordination is gone and there’s nowhere to hide.</p>

<h3 id="the-pm-as-taste-maker">The PM as taste-maker</h3>

<p>I’m going to put a stake in the ground on what taste actually means, because the term gets thrown around without definition.</p>

<p><strong>Product taste is the ability to look at a candidate solution and predict - accurately, quickly, before users see it - whether it will work.</strong> It’s pattern recognition built from many small, fast cycles of <em>I think this will land → ship → see what happened.</em> It’s the muscle that lets you reject nine of ten AI-generated options in five minutes and explain why the tenth wins. It’s not vision. Vision points at a horizon. Taste evaluates an artifact in front of you, now.</p>

<p>You build taste the way Aakash Gupta describes: by evaluating a high volume of prototypes, fast, with stakes attached.<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote" rel="footnote">5</a></sup> A PM who reviews fifteen prototypes a week develops taste faster than a PM who reviews one spec a month. Speed creates compounding judgment — every cycle is a feedback loop, and the loops compound.</p>

<p>The risk worth naming: taste built without user contact becomes preference dressed up as judgment. Real taste stays calibrated against real users. Without that calibration, you’re not exercising taste; you’re exercising opinion.</p>

<h3 id="the-pm-as-responsible-ai-owner">The PM as responsible AI owner</h3>

<p>This is the part of the role I find most under-discussed and most consequential.</p>

<p>When your product ships AI-powered recommendations, automated decisions, or generative outputs, the PM owns the failure modes. Not legal. Not the ethics committee. The PM who shipped it. That includes understanding where models hallucinate, how bias enters training data, what transparency means for your specific users, and when to pump the brakes on a feature that technically works but shouldn’t ship.</p>

<p>Marty Cagan recently wrote that “most product managers will be expected to understand how the enabling AI technology works, what the range of risks involved are, and the work required to mitigate them.”<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">6</a></sup> That’s the right framing. The reality I’ve seen is messier — most PMs aren’t there yet, and the gap between “ships AI features” and “owns AI failure modes” is where a lot of bad product decisions are quietly being made.</p>

<p>This responsibility wasn’t in any PM job description in 2022. It’s in most of them now.</p>

<h2 id="what-this-means-for-the-profession">What this means for the profession</h2>

<p>I want to be direct about implications.</p>

<p>If most of your time goes to documentation, status updates, planning facilitation, and synthesizing information for other people to act on — that’s a role under pressure. Not in five years. Now.</p>

<p>If you’re early in your career, the traditional junior PM apprenticeship is narrowing. The entry-level tasks that used to count as learning the craft are automating. LinkedIn’s APM-to-Product-Builder shift is the institutional signal. What earns a seat at the table now is proof of taste and judgment, not proof of process compliance.</p>

<p>If you’re wondering what to do instead: build things. Use the tools available — prototype something, ship a small product end to end, evaluate fifteen things this week instead of one. The old APM value was learning process through repetition. The new value is developing judgment, and you develop judgment by making hundreds of small product decisions, not by facilitating someone else’s.</p>

<p>Ant Murphy’s 2026 survey of PMs found that 59% rank strategy and business acumen as the most important skills for the next two-to-three years.<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">7</a></sup> Not communication. Not execution. Not collaboration. Strategy. The profession knows what’s coming.</p>

<h2 id="back-to-the-bottleneck">Back to the bottleneck</h2>

<p>If the bottleneck has moved to judgment, the implication for any PM reading this is simple. The job is no longer to coordinate the work of building. The job is to make the decisions that determine whether the building was worth doing.</p>

<p>That sounds obvious until you look at how most PM time is actually spent. Most PMs I know — including, on bad weeks, me — still spend the majority of their hours on synthesis, alignment, and status. The work that maps to the new bottleneck — evaluating prototypes, sharpening point of view, owning failure modes — gets squeezed into whatever time is left. The shift the role demands isn’t a new skill stack. It’s a reallocation of where the hours go.</p>

<p>For PMs who lead with judgment, taste, and curiosity about what users actually need, this is the most exciting time the role has ever offered. The coordination overhead is lifting. What remains is the part that was always supposed to be the point.</p>

<p>“The builders — the people who actually think, who create, who have opinions,” Singhal says, “there’s a renaissance coming for them.”</p>

<p>I believe him. I’m building for it.</p>

<hr />

<p><em>The harder question I’d love your take on: which of the five specialization renames feels most wrong to you, and what would you call it instead?</em></p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>Andrew Ng on the PM bottleneck: see <a href="https://hackernoon.com/andrew-ng-product-team-ratios-evolving-to-just-one-software-developer-for-every-two-product-manager">Hackernoon coverage</a>. <em>(Note: replace with primary source if you can locate Ng’s original talk or post.)</em> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>Cat Wu, <a href="https://www.lennysnewsletter.com/p/how-anthropics-product-team-moves">“How Anthropic’s product team moves faster than anyone else,” Lenny’s Podcast</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p>Nikhyl Singhal, <a href="https://www.lennysnewsletter.com/p/why-half-of-product-managers-are-in-trouble">“Why half of product managers are in trouble,” Lenny’s Podcast</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:4" role="doc-endnote">
      <p>Jenny Wen, <a href="https://www.lennysnewsletter.com/p/the-design-process-is-dead">“The design process is dead. Here’s what’s replacing it,” Lenny’s Podcast</a> <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:5" role="doc-endnote">
      <p>Aakash Gupta, <a href="https://www.news.aakashg.com/p/taste-at-speed">“There’s a New PM Skill. It’s Called Taste at Speed”</a> <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:6" role="doc-endnote">
      <p>Marty Cagan, <a href="https://www.svpg.com/ai-product-management-2-years-in/">“AI Product Management 2 Years In”</a>, Silicon Valley Product Group <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:7" role="doc-endnote">
      <p>Ant Murphy, <a href="https://antmurphy.medium.com/how-product-is-changing-in-2026-78a08f150aca">“How Product is Changing in 2026”</a> <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="AI" /><category term="product management" /><category term="artificial intelligence" /><category term="product manager role" /><category term="AI era" /><summary type="html"><![CDATA[Two years ago I made six predictions about how the PM role would change. I got some right. But the one I missed completely might be the most important one of all.]]></summary></entry><entry><title type="html">Success and Happiness - Two Sides of the Same Coin?</title><link href="https://www.artkreimer.com/Success-vs-Happiness/" rel="alternate" type="text/html" title="Success and Happiness - Two Sides of the Same Coin?" /><published>2025-03-02T00:00:00+00:00</published><updated>2025-03-02T00:00:00+00:00</updated><id>https://www.artkreimer.com/Happiness-and-Success</id><content type="html" xml:base="https://www.artkreimer.com/Success-vs-Happiness/"><![CDATA[<p>What makes someone truly successful? Is it wealth, recognition, or something deeper? Recently, I reread <strong>Outliers</strong> by Malcolm Gladwell, a book that explores the factors behind extraordinary achievement. I first read it about ten years ago, and many of its key points stayed with me - especially the famous 10,000-hour rule. This time, however, what stood out most was the story of Roseto, a small town in the U.S. with unusually low rates of cardiovascular disease. What made Roseto residents healthier wasn’t genetics, exercise, or diet - researchers found that the town’s unique culture and strong sense of community were the key factors behind this phenomenon.</p>

<p>This story resonated with me on a personal level. It made me think about the times in my life when strong relationships and a sense of belonging had the most profound impact on my wellbeing far beyond any career achievement or milestone. It reinforced the idea that, as social beings, we need friends, family, and community especially as we get older and not only to survive but to thrive. The Roseto example reminded me of the <strong>Harvard Study of Adult Development</strong>, one of the longest-running studies on happiness. Its key finding? <strong>Relationships and social connections are the most significant predictors of long-term happiness and well-being</strong>. Quality mattered more than quantity - close, supportive relationships contributed to emotional resilience, physical health, and overall life satisfaction.</p>

<p>During a discussion with some friends about the book, I mentioned how the Roseto example stood out to me because it highlighted the importance of building a circle of support and nurturing deep, lasting relationships - something that feels more critical to me at this stage of my life than simply chasing success. One of my friends challenged me, saying, “Why is that? Success and happiness aren’t separate for me - they go hand in hand. I wouldn’t consider something a success if it didn’t bring me happiness. Don’t you see it the same way?”</p>

<p>That question made me reflect. Success and happiness are often intertwined, but they aren’t necessarily the same. Consider someone like <strong>Kurt Cobain</strong>, Nirvana frontman, or <strong>Chester Bennington</strong>, lead vocalist of Linkin Park, both widely regarded as successful, yet they struggled with inner turmoil and eventually took thier own life. Their stories highlight how external success doesn’t always translate to personal fulfillment.</p>

<p>We tend to define success through external achievements - career milestones, financial stability, or recognition. But my friend saw success as something that also had to bring happiness. He also pointed out that his definition of success had changed over time: when he was younger, success meant getting a great job and building a family; later, it became about raising great kids; and now, it’s tied to their success and well-being.</p>

<p>When I think about it, I still see them as two different things, and here’s why: there are many highly successful people who are not happy. Does this mean they set the wrong optimization function? Or are success and happiness just separate? My friend’s argument - “Can you actually consider someone successful if they’re not happy?” - made me pause. My answer is yes.</p>

<p>There are many ways to measure success: career, wealth, family, fame, impact. But happiness is different. Naval Ravikant talks about this distinction. He believes success comes from doing what you love and are uniquely good at, while happiness is a skill - something to be cultivated. True happiness, according to Naval, comes from removing the constant sense that something is missing. It’s about inner peace, managing desires, and detaching from the endless pursuit of external validation.</p>

<p>Naval’s perspective is particularly relevant because he challenges the traditional definitions of success and happiness. He argues that success is an external pursuit, while happiness is an internal state—one that can be cultivated regardless of external achievements. A few of his quotes stand out:</p>

<blockquote>
  <p>“Happiness is what’s there when you remove the sense that something is missing in your life.”<br />
 “Happiness is being satisfied with what you have. Success comes from dissatisfaction. Choose.”<br />
 “Happiness = Health, Wealth, and Good Relationships.”</p>
</blockquote>

<p>At this point in my life, I see success as multifaceted. A person can be successful in one domain but still be unhappy if they lack inner peace. However I like the idea of integrating both - measuring my personal success not just by wealth, career, or relationships, but also by my happiness and satisfaction in these areas.</p>

<p>What about you? Do you see success and happiness as separate, or do they always go together? Looking back at Roseto, their success wasn’t measured by wealth or status, but by the strength of their relationships. Could it be that true success is not what we achieve individually but what we cultivate within our communities?</p>]]></content><author><name>Art Kreimer</name></author><category term="Books Review" /><category term="Books" /><summary type="html"><![CDATA[Balancing external accomplishments with inner peace. If success doesn't bring happiness, is it really success?]]></summary></entry><entry><title type="html">6 ways the role of a product manager is going to change in the next decade</title><link href="https://www.artkreimer.com/6_ways_pm_role_will_change/" rel="alternate" type="text/html" title="6 ways the role of a product manager is going to change in the next decade" /><published>2024-04-10T00:00:00+00:00</published><updated>2024-04-10T00:00:00+00:00</updated><id>https://www.artkreimer.com/6-ways-pm-role-will-change</id><content type="html" xml:base="https://www.artkreimer.com/6_ways_pm_role_will_change/"><![CDATA[<figure style="width: 50%" class="align-right">
  <img src="/assets/images/feature_row/futurefest_mainpage.jpg" alt="ScotiaBank Digital - Future Fest Craft 2024 Magazine" />
</figure>

<p>For the last several years, Scotia Digital’s Product and Design Communities of Practice have hosted the annual Future Fest Craft festival, a two-day conference focused on craft and innovation. I’ve been honored to present and participate in panel discussions on “Crafting the Future with AI” for the past two years. Last year, I explored what it takes to build AI-powered features, while this year, my co-host and I led a workshop on leveraging GenAI, specifically Microsoft Copilot, to boost PM productivity during discovery, ideation, and communication.</p>

<p>Along with the two-day festival, we also created a printed magazine this year. I was lucky enough to contribute to the first edition of the Future Fest Craft Magazine. Here is my article in full.</p>

<h2 id="decoding-the-next-decade-6-ways-the-product-manager-role-will-change">“Decoding the next decade: 6 ways the product manager role will change”</h2>

<p>Product managers are the bridge between business goals, design vision, and technical feasibility. They define the product’s purpose (the <strong>“what”</strong> and <strong>“why”</strong>) and collaborate with their team to determine the <strong>“how”</strong> to make it a reality. Customer champions at heart, they understand user needs and translate them into valuable solutions that benefit both users and the company.</p>

<p>The craft of product management requires many different skills. Great product managers are powerful communicators with high emotional intelligence, allowing them to influence stakeholders and navigate complex situations. They are also experts in their domains, keeping a pulse on industry trends to make strategic investments that maximize product value.</p>

<p>They don’t just deliver features. They deliver impactful outcomes. Their focus is on prioritizing the right features to ensure the product solves real problems and creates significant value. It’s about shipping the right software, not just any software.</p>

<p><strong>The future of product management</strong></p>

<p>As we look to the future, we find ourselves in an era of rapid technological advancements, shifting consumer demands, and ever-increasing competition. The role of a product manager is poised for significant change in the next decade, driven by these advancements and evolving business landscapes.</p>

<p>Here are 6 ways the role of a product manager is going to change:</p>

<ol>
  <li>
    <p><strong>AI-powered assistance:</strong> Artificial intelligence (AI) will become a product manager’s collaborator. AI assistants like Microsoft’s Copilot or OpenAI’s ChatGPT will automate repetitive tasks like data analysis, competitor research, and generating user personas. This will free up time for product managers to focus on higher-level strategy and creative problem-solving.</p>
  </li>
  <li>
    <p><strong>Product managers get AI-savvy:</strong> You can expect AI capabilities to become the norm within software products. This trend will make machine learning (ML)  developers an essential part of product development teams. For product managers to excel, a baseline understanding of ML concepts will be crucial, which include grasping how algorithms work, the probabilistic nature of AI-powered products, and the ethical implications of AI-powered product features.</p>
  </li>
  <li>
    <p><strong>Customer-centricity on steroids:</strong> The focus on customer-centricity will intensify. Product managers will need to become masters of user experience research and design thinking. Understanding user pain points, frustrations, and needs will be paramount to creating products that resonate deeply with customers.</p>
  </li>
  <li>
    <p><strong>The rise of the product visionary:</strong> In a world with ever-increasing competition and faster innovation cycles, product managers will need to be more visionary than ever. They’ll need to anticipate future trends, identify unmet customer needs, and translate them into a compelling product roadmap.</p>
  </li>
  <li>
    <p><strong>Collaboration across boundaries:</strong> The siloed approach to software development will fade away. Product managers will need to collaborate effectively with cross-functional teams like engineering, marketing, business, and sales. Strong communication and interpersonal skills will be of most importance for navigating diverse perspectives and achieving product goals.</p>
  </li>
  <li>
    <p><strong>Adaptability and lifelong learning:</strong> The pace of technological change will only accelerate. Product managers will need to be lifelong learners, constantly honing their skills and staying up to date about emerging technologies like AI, blockchain, quantum computing, and the Internet of Things.</p>
  </li>
</ol>

<p>The future holds exciting challenges and opportunities for product managers. Their role will continue to be indispensable, serving as the driving force behind successful product strategies, seamless customer experiences, and sustainable business growth.</p>

<p>A product manager’s focus on customer needs and strategic vision will become even more vital in the next decade.</p>

<p>Technological advancements like AI, changing customer expectations, and the need for cross-team collaboration will reshape the role. To succeed, product managers must become adept at using AI tools, understanding ML concepts, mastering customer-centric design, developing compelling visions, collaborating across teams, and embracing adaptability and lifelong learning.</p>

<p>Let’s embrace the future, combining our experience with a relentless drive for innovation and excellence to develop and ship great products to customers.</p>

<p>What change are you most excited to embrace as a product manager?</p>]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="PM" /><summary type="html"><![CDATA[My predictions on how the product manager role is changing and how you can adapt to build the products of the future.]]></summary></entry><entry><title type="html">Humans and AI: Balancing AI Innovation with Human Skills</title><link href="https://www.artkreimer.com/humans_are_underrated/" rel="alternate" type="text/html" title="Humans and AI: Balancing AI Innovation with Human Skills" /><published>2024-01-07T00:00:00+00:00</published><updated>2024-01-07T00:00:00+00:00</updated><id>https://www.artkreimer.com/humans-are-underrated</id><content type="html" xml:base="https://www.artkreimer.com/humans_are_underrated/"><![CDATA[<p>As we navigate the rapidly evolving AI landscape, concerns about its impact on our lives and jobs are at an all-time high. This post was written mid-last year, and based on a presentation I gave in response to significant concerns about AI taking over tech jobs, especially following the release of ChatGPT. Last week, during a conversation with an advisor at my bank, I mentioned that I am working on introducing new AI-based features into Scotiabank’s digital products. Her strong reaction, expressing fear about AI knowing too much and posing a threat, prompted me to revisit this topic and finally publish this blog post.</p>

<p>I have always been a passionate techno-optimist<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, particularly about the huge AI potential. While the doomed future portrayed in Terminator was entertaining and raising some concerns about the future of humanity, I genuinely believe in AI’s immense potential to benefit our world. Recent revelations by Geoffrey Hinton<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>, whom I highly admire, have tempered my pure optimism, moving me towards a position of cautious optimism or pragmatic realism. However, I am far from being an AI doomer.</p>

<p>Though I believe AI practitioners should reinforce guardrails and closely monitor our journey toward true AGI, we are still quite far away from achieving it. I see AI as a catalyst for human progress and an augmentation of human intellect. Andrew Ng famously said “AI is a new electricity”<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> and indeed like electricity in 19th century, AI is changing the way we live and work in 21st century. It changes all aspects of our lives from self-driving cars to advanced medical diagnosis. While the transformative power of AI is exciting, it’s crucial to acknowledge some of the challenges that come with it.</p>

<p>Beyond doomsday predictions, a significant concern is AI’s impact on the job market and the potential elimination or transformation of certain jobs. Unfortunately it’s true - in the future some professions will become obsolete. Historically, with the introduction of new technologies, people had to shift and change their professions. Telegram operators, phone switchboard operators, manual bookkeepers, and camera film developers are examples of professions that no longer exist, yet society has benefited from technological advancements. And at the same time, these technologies created new jobs. Jobs like Social media managers, Cyber security specialists, VR designers, UX designers didn’t exist 30 years ago. New technologies will always change how we live, which in turn affects our jobs.</p>

<p>Technological change so far was a huge catalyst and brought prosperity to majority of the nations on planet Earth. On a global scale, we live far better lives compared to 100 years ago. We have eradicated many illnesses, the number of people living in poverty and hunger is constantly decreasing. Many advancements can be attributed to technological progress, including our advancements in AI algorithms<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup>. AI is going to augment or even replace some of the jobs, but majority of humans would benefit from these changes.</p>

<p>Let’s delve into what AI algorithms lack and what I believe would be exceptionally challenging for them to replicate. To begin with, AI is essentially an algorithm driven by data and human inputs. Without human data, AI is essentially powerless. Recent studies have shown that training models on AI-generated data can lead to models that struggle to generalize and ultimately collapse<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">5</a></sup>. Therefore, it’s evident that current AI systems can’t generate data to improve or enhance AI models; they rely on human data to become better.</p>

<p>Current AI algorithms excel at processing vast amounts of data, predicting outcomes, generating text, and creating images. But despite its advancements, AI still falls short in certain areas when compared to human capabilities. Here are some key traits where AI currently lacks:</p>

<ol>
  <li><strong>Creativity</strong>: While AI can replicate and optimize existing solutions, humans possess the unique ability to generate completely new ideas and concepts, sometimes without any prior knowledge. The extent of AI’s creative capabilities remains a topic of ongoing debate, largely due to the complex nature of defining ‘Creativity.’ In my view, current AI technologies still lack the essence of genuine creativity.</li>
  <li><strong>Ethics and Morality</strong>: AI lacks the ability for ethical and moral reasoning, a critical aspect in fields such as law, medicine, psychology, and social work. A significant challenge with AI algorithms is their inherent bias, stemming from the data they are trained on, which can lead to serious ethical concerns.</li>
  <li><strong>Adaptability</strong>: Although humans are prone to making mistakes, they are capable of reacting and adapting to a novel situation, in contrast to AI, which may not recognize or respond to new situations outside its training scope. For instance, a self-driving car might struggle with an unfamiliar traffic sign or an unexpected scenario.</li>
  <li><strong>Empathy</strong> and <strong>Emotional Intelligence</strong>: AI is currently unable to truly empathize with humans or fully understand human emotions. While it’s true that some humans may also struggle with these abilities, they are indispensable in fields such as counseling, social work, healthcare, and customer service.</li>
  <li><strong>Contextual Understanding</strong>: AI operates based on the data it receives and, while capable of processing vast quantities of information, often lacks the nuanced ability to comprehend the context and deeper meaning behind that data.</li>
  <li><strong>Common Sense</strong>: Most current AI and ML models are highly specialized and lack the broad common sense inherent to humans, an attribute crucial in decision-making and problem-solving. While AI can generalize to a certain extent, it still falls short of human capabilities, at least for now.</li>
</ol>

<p>So don’t be afraid of AI; embrace it. We are still far from AI replacing us. Utilize AI-based tools to your advantage and contemplate the role AI will play in your profession’s future. In recent years, there has been an explosion of new AI tools on the market. From text generation and image/audio recognition/generation to predictive analytics, these tools are revolutionizing the way businesses operate and providing new opportunities for innovation and growth. If you need to create content, explore platforms like ChatGPT, Copy.ai, and Jasper. As a software developer, consider trying tools such as Codex, Github Copilot, or ChatGPT to learn or bootstrap new algorithms. Looking for a new image for your blog? Check out DALL-E, Stable Diffusion, or Midjourney. The possibilities are endless.</p>

<p>Embrace the power of AI and commit to constant learning and adaptation. Yuval Noah Harari, the author of “21 Lessons for the 21st Century,” predicts that jobs will change every 10 or 15 years. As a result, we will need to reinvent ourselves professionally several times during our lifetime. Maintain a curious and open mindset and keep on learning.</p>

<p>In conclusion, while AI undoubtedly presents challenges, it also offers immense opportunities for innovation, efficiency, and progress. We need to approach it with an open mind, balancing optimism with pragmatism. By understanding both the capabilities and limitations of AI, we can harness its power in a responsible and creative manner. Let’s embrace AI as a tool for growth and transformation, always remembering that the true essence of innovation lies in the human spirit – something beyond AI’s capability to replicate. Together, we can navigate this new era, blending technology with the unique and irreplaceable qualities of human creativity, empathy, and insight. What are your thoughts on how we can best navigate and shape the future with AI?</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>I share a lot of Marc Andreessen’s views, I am just a bit less optimistic and more cautious. Check out these two posts <a href="https://a16z.com/the-techno-optimist-manifesto/">The Techno-Optimist Manifesto</a> and <a href="https://a16z.com/ai-will-save-the-world/">Why AI Will Save the World</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p><a href="https://mitsloan.mit.edu/ideas-made-to-matter/why-neural-net-pioneer-geoffrey-hinton-sounding-alarm-ai">Why neural net pioneer Geoffrey Hinton is sounding the alarm on AI</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p><a href="https://www.wsj.com/video/andrew-ng-ai-is-the-new-electricity/56CF4056-4324-4AD2-AD2C-93CD5D32610A">Andrew Ng: AI Is the New Electricity</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:4" role="doc-endnote">
      <p>Great article on numerous advances humanity made in 2023 <a href="https://futurecrunch.com/goodnews2023/">66 Good News Stories You Didn’t Hear About in 2023</a> <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:6" role="doc-endnote">
      <p>Arxiv research paper <a href="https://arxiv.org/abs/2305.17493v2">The Curse of Recursion: Training on Generated Data Makes Models Forget</a> <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Art Kreimer</name></author><category term="AI/ML" /><category term="Gen AI" /><category term="AI" /><summary type="html"><![CDATA[My thoughts on transformative power of AI and its impact on job dynamics and the balance between technological and human skills.]]></summary></entry><entry><title type="html">AI Product Development: A Deep Dive into Key Considerations</title><link href="https://www.artkreimer.com/AI-primer-for-pm-part2/" rel="alternate" type="text/html" title="AI Product Development: A Deep Dive into Key Considerations" /><published>2023-09-09T00:00:00+00:00</published><updated>2023-09-09T00:00:00+00:00</updated><id>https://www.artkreimer.com/AI-primer-for-pm-part2</id><content type="html" xml:base="https://www.artkreimer.com/AI-primer-for-pm-part2/"><![CDATA[<p class="notice--info"><em>This is Part II of the AI for PMs series. See <a href="/AI-primer-for-pm/">AI for PMs series Part I</a> on AI use cases and what to do before you even start working on AI product.</em></p>

<h2 id="introduction">Introduction</h2>

<p>In <a href="/AI-primer-for-pm/">AI for PMs Part I</a>, I discussed the history of Artificial Intelligence, the diversity of applications and AI use cases, and highlighted key considerations to keep in mind before embarking on the development of an AI product. In this part, I will delve into crucial aspects to be mindful of during the development of an AI product. I will address the following topics, provide examples, and suggest potential mitigation strategies:</p>

<ul>
  <li>Bias and Unfairness</li>
  <li>Explainability</li>
  <li>Privacy, Compliance, and Security</li>
  <li>Continuous Evaluation</li>
</ul>

<p>Now, let’s explore each of these areas in detail.</p>

<h3 id="bias-and-unfairness">Bias and Unfairness</h3>
<p>AI algorithms can perpetuate or even exacerbate existing biases in society. This can happen when algorithms are trained on biased datasets or if their structures inadvertently introduce bias.</p>

<p><strong>Examples:</strong></p>

<ol>
  <li><strong>Gender Bias in Resume Screening</strong>: Algorithms that favour resumes with male-dominated activities can perpetuate gender imbalances in certain industries.</li>
  <li><strong>Racial Bias in Facial Recognition</strong>: These systems may struggle to accurately identify individuals from diverse ethnic backgrounds, leading to potential injustices.</li>
</ol>

<p><strong>Mitigation Strategies:</strong></p>
<ul>
  <li>Employ diverse training datasets.</li>
  <li>Perform algorithmic fairness audits.</li>
  <li>Involve a diverse team in the development and validation process.</li>
</ul>

<h3 id="explainability">Explainability</h3>
<p>The “black box” nature of complex AI algorithms often hinders transparency, making explaining model decisions and outcomes difficult. In some highly regulated industries like healthcare and banking it could be a real issue.</p>

<p><strong>Examples:</strong></p>
<ol>
  <li><strong>Healthcare Diagnostics</strong>: If a machine learning model predicts a particular diagnosis, clinicians must understand why so they can make informed decisions.</li>
  <li><strong>Loan Approval Systems</strong>: Knowing why a loan application was rejected can help applicants improve their financial behaviour.</li>
</ol>

<p><strong>Mitigation Strategies:</strong></p>
<ul>
  <li>Use explainable AI models or techniques like LIME or SHAP.</li>
  <li>Provide clear documentation for internal algorithms and decision-making processes.</li>
</ul>

<h3 id="privacy-compliance-and-security">Privacy, Compliance, and Security</h3>
<p>Ensuring user privacy and compliance with data protection regulations is essential. This includes safeguarding user data through techniques like data anonymization and encryption.</p>

<p><strong>Examples:</strong></p>
<ol>
  <li><strong>Healthcare Data</strong>: Personal health information should be anonymized before being processed by AI models.</li>
  <li><strong>Financial Transactions</strong>: Protecting users’ banking and credit card information using encryption and secure channels is paramount.</li>
</ol>

<p><strong>Mitigation Strategies:</strong></p>
<ul>
  <li>Adhere to GDPR or similar data protection regulations.</li>
  <li>Use privacy-preserving techniques like differential privacy.</li>
</ul>

<h3 id="continuous-evaluation">Continuous Evaluation</h3>
<p>Even more than any regular software products, AI systems require ongoing monitoring and assessment to ensure they meet desired performance criteria and remain relevant. Without the most recent data, AI model performance might deteriorate.</p>

<p><strong>Examples</strong>:</p>
<ol>
  <li><strong>Real-Time Fraud Detection</strong>: The AI model must adapt and improve as fraudsters change tactics.</li>
  <li><strong>E-commerce Recommendation Systems</strong>: As product inventory or consumer behaviour changes, the recommendation algorithms should evolve.</li>
</ol>

<p><strong>Mitigation Strategies</strong>:</p>
<ul>
  <li>Work with your Data Scientists and ML Engineers to develop a plan to continuously monitor model performance and create a continuous data feed to ensure the model has all the latest data needed.</li>
  <li>Regularly collect user feedback and update the model as needed.</li>
</ul>

<p>You can build more ethical, efficient, and effective AI systems by being attentive to the aforementioned areas.</p>

<h2 id="conclusion">Conclusion</h2>

<p>AI offers numerous opportunities for innovation in product management. Being aware of its limitations can significantly improve the outcomes of your projects and make you a better Product Manager.</p>

<p>Be curious, follow AI trends, learn and create products that solve customer pain points and delight customers with the help of AI/ML features and algorithms.</p>

<h2 id="additional-resources">Additional Resources</h2>

<p>This article is just a primer to provide some knowledge and kick off your curiosity to learn more. Here is a short list of articles, courses and books to keep you going:</p>

<ul>
  <li><a href="https://www.linkedin.com/learning/becoming-an-ai-first-product-leader/becoming-an-ai-first-product-leader?autoplay=true&amp;u=2142274">Becoming an AI-First Product Leader (LinkedIn Learning)</a></li>
  <li><a href="https://www.linkedin.com/learning/artificial-intelligence-for-business-leaders/welcome-to-the-course?autoplay=true&amp;u=2142274">Artificial Intelligence for Business Leaders (LinkedIn Learning)</a></li>
  <li><a href="https://www.coursera.org/specializations/ai-product-management-duke">AI Product Management Specialization (Coursera)</a></li>
</ul>

<div style="text-align: center;">
<iframe src="https://artkreimer.substack.com/embed" title="Newspaper sign up" height="320" width="400" style="border:1px solid #EEE; background:white;" frameborder="0" scrolling="no"></iframe>
</div>]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="AI" /><category term="Product Management" /><summary type="html"><![CDATA[Learn how to navigate critical issues like bias, black-box nature of some AI algorithms, data security, and why we need to contantly evaluate ML models in production.]]></summary></entry><entry><title type="html">Beyond the Buzz: AI primer for Product Managers</title><link href="https://www.artkreimer.com/AI-primer-for-pm/" rel="alternate" type="text/html" title="Beyond the Buzz: AI primer for Product Managers" /><published>2023-09-03T00:00:00+00:00</published><updated>2023-09-03T00:00:00+00:00</updated><id>https://www.artkreimer.com/AI-primer-for-pm</id><content type="html" xml:base="https://www.artkreimer.com/AI-primer-for-pm/"><![CDATA[<p class="notice--info"><em>This is Part I of the AI for PMs series. See <a href="/AI-primer-for-pm-part2/">AI for PMs - Part II</a> on what to pay attention to when you start building an AI product.</em></p>

<h2 id="introduction">Introduction</h2>

<p>Today, AI / ML is at the cutting edge of technological innovation, impacting various sectors from healthcare to finance. Since OpenAI launched ChatGPT late last year, terms like LLMs, GPT, and GenAI have become buzzwords. Product hunt is filled with GenAI-based products, and the AI product landscape is growing exponentially <sup id="fnref:0" role="doc-noteref"><a href="#fn:0" class="footnote" rel="footnote">1</a></sup>. This article aims to clarify some misconceptions about AI and ML and what Product Managers need to know about them. While AI isn’t a cure-all, understanding its capabilities is key to competitive advantage.</p>

<p>It’s worth noting that the concept of AI is not new; it has ancient origins and has been part of human imagination for centuries, appearing in myths like the Biblical Golem and ancient automatons. The term ‘Artificial Intelligence’ itself was coined in 1956<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">2</a></sup> and has been a subject of academic and cultural discussion ever since. With over 70 years of research history, AI and ML have been employed across various applications, but only in the last couple of decades, advancements in technology, including the explosion of internet usage and vast amounts of data, the availability of powerful cloud computing, as well as specialized GPUs and AI chipsets, have catalyzed the evolution and improvements of ML models.</p>

<figure class="align-center">
  <img src="/assets/images/aiml_google_ngrams.jpg" alt="AI and ML trend on Google Ngrams" />
  <figcaption>AI and ML trend on Google Ngrams</figcaption>
</figure>

<h2 id="so-what-is-ai">So, what is AI?</h2>

<p>AI is an umbrella term covering various subfields, including Machine Learning (ML), Deep Learning, Generative AI (GenAI), and the ultimate goal, Artificial General Intelligence (AGI). Let’s go over the definitions:</p>

<dl>
  <dt>Artificial Intelligence (AI)</dt>
  <dd>Computer systems capable of performing tasks that typically require human intelligence.</dd>
  <dt>Machine Learning (ML)</dt>
  <dd>A subset of AI focused on developing algorithms that enable computers to make decisions or predictions.</dd>
  <dt>Deep Learning</dt>
  <dd>Specialized area within ML that utilizes multi-layer artificial neural networks to learn from large data sets.</dd>
  <dt>Generative AI (GenAI)</dt>
  <dd>Deep learning models capable of generating new data or content like images, music, video, or text.</dd>
  <dt>Artificial General Intelligence (AGI)</dt>
  <dd>The ultimate goal in the AI spectrum—a fully autonomous system capable of human-level intelligence across various tasks without requiring specialized programming.</dd>
</dl>

<h2 id="types-of-ml">Types of ML</h2>

<p>There are four main types of ML models:</p>

<ol>
  <li>
    <p><strong>Supervised Learning</strong>: The model is trained on labelled data to make predictions or decisions.</p>

    <p>Applications: Credit score applications, email spam detection, news topic classifications, and image recognition.</p>
  </li>
  <li>
    <p><strong>Unsupervised Learning</strong>: The model uses unlabeled data to find patterns or groupings.</p>

    <p>Applications: Document clustering, customer segmentation based on buying behaviour, anomaly detection.</p>
  </li>
  <li>
    <p><strong>Semi-supervised Learning</strong>: The model uses both labelled and unlabeled data for training, often to improve performance.</p>

    <p>Applications: Image recognition and other supervised learning applications.</p>
  </li>
  <li>
    <p><strong>Reinforcement Learning</strong>: The model learns to make decisions by interacting with an environment to achieve a goal or maximize some notion of cumulative reward.</p>

    <p>Applications: Game playing, Autonomous vehicles.</p>
  </li>
</ol>

<p>Now, let’s dive into different AI/ML use cases.</p>

<h2 id="aiml-use-cases">AI/ML Use Cases</h2>

<p>Artificial Intelligence (AI) and Machine Learning (ML) technologies are revolutionizing diverse industries. Here are some notable use cases and applications where AI/ML algorithms are making a difference:</p>

<ul>
  <li><strong>Credit Scoring &amp; Anomaly Detection</strong>: Often use Supervised and Unsupervised Learning models.</li>
  <li><strong>Chatbots &amp; Customer Service &amp; Customer Insights</strong>: Primarily rely on NLP, which can be built using Supervised Learning.</li>
  <li><strong>Stock Trend Analysis</strong>: Typically use Time Series Algorithms, a form of Supervised Learning.</li>
  <li><strong>Content Generation</strong>: Generative AI models to generate text, audio, code and video using form of Deep Learning that uses both Supervised and Unsupervised ML models,</li>
  <li><strong>Supply Chain Optimization</strong>: Utilizes Optimization Algorithms, often built using Reinforcement Learning.</li>
  <li><strong>Recommendation Systems</strong>: Generate individualized product, service, or content suggestions commonly employed in e-commerce, streaming services, and social media to boost user engagement. Both supervised and unsupervised models are being used.</li>
</ul>

<h2 id="key-considerations-before-embarking-on-an-ai-product-journey">Key Considerations Before Embarking on an AI Product Journey</h2>
<p>Multiple critical factors have to be considered before you dive into developing an AI-powered product. Understanding these can be the difference between a successful launch and a missed opportunity.</p>

<p>Here are some pivotal elements to keep in mind:</p>

<p><strong>Identifying AI Opportunities</strong> - Make sure the AI solution addresses a genuine customer need rather than merely finding an excuse to use an advanced AI model.</p>
<ul>
  <li>Conduct market research to identify gaps or pain points that AI can solve.</li>
  <li>Validate the idea with stakeholders and potential users.</li>
  <li><strong>“Fall in love with the problem and not with the solution”</strong> - Don’t implement AI for the sake of adding AI to your product.</li>
</ul>

<p><strong>Finding the Right Partner</strong> - Developing AI/ML applications requires specialized expertise. Finding the right partner is crucial whether you look inside your organization or seek an external vendor.</p>
<ul>
  <li>Evaluate potential partners based on their previous work and their expertise in the domain.</li>
  <li>Ensure that the partner aligns with your organization’s culture and goals.</li>
</ul>

<p><strong>Data Is Key</strong> - If you have to build your model, the right data and its quality would be of the utmost importance. Data is the backbone of any AI project and is crucial for training models. In many cases, continuous training might also be necessary to adapt to new data patterns.</p>

<ul>
  <li>Ensure you have access to the right amount of a high-quality and relevant data.</li>
  <li>Consider the ethical implications and legal requirements associated with data collection and usage, more about it in my next <a href="/AI-primer-for-pm-part2">post</a>.</li>
</ul>

<p><strong>AI-Application Fit</strong> - Understanding the pros and cons of various AI and ML techniques is essential. Discuss with your partner to choose the most appropriate algorithm for your specific problem and use case.</p>

<ul>
  <li>Perform a feasibility analysis to identify the most effective AI or ML techniques.</li>
  <li>Keep scalability, cost, and performance in mind when choosing algorithms.</li>
</ul>

<p><strong>Resource Allocation</strong> - Be clear about the human and computational resources required to bring your AI project to life.</p>
<ul>
  <li>Create a detailed project timeline and budget.</li>
  <li>Consider the long-term maintenance costs of the AI system.</li>
</ul>

<h2 id="conclusion">Conclusion</h2>
<p>AI offers numerous opportunities for innovation in product management. Awareness of its possibilities can significantly benefit your role as a Product Manager. Be curious, follow AI trends, learn and create products that solve customer pain points and delight customers with the help of AI/ML features and algorithms.</p>

<p>Check <a href="https://www.artkreimer.com/AI-primer-for-pm-part2/">AI for PMs - Part 2</a> for tips on what to pay attention to when building AI-based products.</p>

<h2 id="additional-resources">Additional Resources</h2>

<p>This article is just a primer to provide some knowledge and kick off your curiosity to learn more. Here is a short list of articles, courses and books to keep you going:</p>

<ul>
  <li><a href="https://www.coursera.org/learn/ai-for-everyone?index=prod_all_products_term_optimization.&amp;utm_medium=sem&amp;utm_source=gg&amp;utm_campaign=B2C_NAMER_ibm-data-science_ibm_FTCOF_professional-certificates_country-US-country-CA-pmax-nonNRL-within-14d&amp;campaignid=19995348162&amp;adgroupid=&amp;device=c&amp;keyword=&amp;matchtype=&amp;network=x&amp;devicemodel=&amp;adposition=&amp;creativeid=&amp;hide_mobile_promo&amp;gclid=CjwKCAjw0ZiiBhBKEiwA4PT9z0U5-OHPBbnDbxNJzDXMK3G1ff6KOsrNn0v7duy_IC1ik_knIc1EKBoCX34QAvD_BwE">AI for Everyone (Coursera)</a></li>
  <li><a href="https://www.linkedin.com/learning/becoming-an-ai-first-product-leader/becoming-an-ai-first-product-leader?autoplay=true&amp;u=2142274">Becoming an AI-First Product Leader (LinkedIn Learning)</a></li>
  <li><a href="https://www.linkedin.com/learning/artificial-intelligence-for-business-leaders/welcome-to-the-course?autoplay=true&amp;u=2142274">Artificial Intelligence for Business Leaders (LinkedIn Learning)</a></li>
  <li><a href="https://www.coursera.org/specializations/ai-product-management-duke">AI Product Management Specialization (Coursera)</a></li>
  <li>Generative AI Landscape (<a href="https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/">Sequoia</a> ; <a href="https://www.antler.co/blog/generative-ai">Antler</a>)</li>
  <li><a href="https://www.evidentlyai.com/ml-system-design">A collection of a real-world ML applications</a></li>
  <li>Prompt Engineering Guide: <a href="http://Promptingguide.ai">Promptingguide.ai</a></li>
  <li>Keep up with the latest news in the AI space with <a href="https://tldr.tech/ai">TLDR AI</a></li>
</ul>

<div style="text-align: center;">
<iframe src="https://artkreimer.substack.com/embed" title="Newspaper sign up" height="320" width="400" style="border:1px solid #EEE; background:white;" frameborder="0" scrolling="no"></iframe>
</div>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:0" role="doc-endnote">
      <p>AI and GenAI landscape is enormous. Here are some examples: <a href="https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/">Sequoia GenAI</a>, <a href="https://www.datacamp.com/cheat-sheet/the-generative-ai-tools-landscape">DataCamp</a> <a href="#fnref:0" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:1" role="doc-endnote">
      <p>The field of <a href="https://en.wikipedia.org/wiki/Artificial_intelligence">AI research</a> was founded at a <a href="https://en.wikipedia.org/wiki/Dartmouth_workshop">workshop</a> held on the campus of <a href="https://en.wikipedia.org/wiki/Dartmouth_College">Dartmouth College</a>, USA during the summer of 1956. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="AI" /><category term="Product Management" /><summary type="html"><![CDATA[Demystify AI and ML for Product Managers — from supervised learning to GenAI use cases and what to consider before building your first AI product.]]></summary></entry><entry><title type="html">How to build a resume chatbot using the power of LLMs</title><link href="https://www.artkreimer.com/How-To-Build-Resume-Bot-powered-by-llm/" rel="alternate" type="text/html" title="How to build a resume chatbot using the power of LLMs" /><published>2023-09-02T00:00:00+00:00</published><updated>2023-09-02T00:00:00+00:00</updated><id>https://www.artkreimer.com/How-To-Build-Resume-Bot-powered-by-llm</id><content type="html" xml:base="https://www.artkreimer.com/How-To-Build-Resume-Bot-powered-by-llm/"><![CDATA[<h2 id="introduction">Introduction</h2>

<p>In my previous post <a href="https://www.artkreimer.com/How-to-Analyze-App-Reviews-Using-GPT/">“Exploring the Power of LLMs for NLP Tasks”</a> I explored how Language Learning Models (LLMs) can be utilized for a range of Natural Language Processing (NLP) tasks —sentiment analysis, topic extraction, content generation, summarization, and more. Building on that foundation, this post aims to guide you through my journey of crafting a resumeGPT chatbot. This chatbot is powered by OpenAI’s GPT-3.5, Langchain, and originally Streamlit — I’ve since migrated it to <a href="https://bolt.new">Bolt</a> for a significantly improved UX.</p>

<p>Check out my ResumeGPT Chatbot at <a href="https://kredar-resumegpt-imp-s3m5.bolt.host/">kredar-resumegpt-imp-s3m5.bolt.host</a> or embedded here <a href="https://www.artkreimer.com/resume/">ResumeGPT Bot</a></p>

<p>Feel free to fork my repo on <a href="https://github.com/kredar/resumeGPT">Github</a></p>

<h2 id="the-motivation-behind-the-chatbot">The Motivation Behind the Chatbot</h2>

<p>I wanted a chatbot that could serve as a personal assistant—one designed to answer questions about my professional experience, educational background, and career aspirations, essentially a resume bot. Last year, I began this project using Google Dialogflow CX. Anyone in the Conversational AI space will concur that crafting an excellent chatbot experience on such platforms can be challenging and time-consuming. Although I had a working prototype, it still required extensive testing and tweaks in both conversational design and intent training. However, the game changed with the release of OpenAI’s GPT-3.5 Turbo model. I pivoted to using large language models (LLMs), resulting in a resume chatbot built with Langchain, and FAISS as a vector store, all showcased through a Streamlit frontend.</p>

<h2 id="overview-of-the-logic-architecture">Overview of the Logic Architecture</h2>

<p>I use a standard architecture commonly used in Retrieval Augmented Generation (RAG) applications. Retrieval-augmented generation (RAG) is an AI framework that combines an information retrieval component with an LLM foundational model for a text generator model. It works by retrieving data that is not present in a foundation model and adding to the prompts to retrieve the relevant data. It allows the foundational model to generate text that it had no knowledge about. It also allows updating data without retraining or finetuning the model, enabling access to the latest information for generating reliable outputs via retrieval-based generation. This approach also helps to reduce hallucinations in GenAI applications and improve the quality of LLM-generated responses.</p>

<p>This architecture is designed to:</p>

<ol>
  <li>Create embeddings for all documents.</li>
  <li>Store these embeddings in a vector store or database for rapid retrieval.</li>
</ol>

<h3 id="data-storage">Data Storage</h3>

<p>My data is stored in a simple CSV file, featuring just two columns <code class="language-plaintext highlighter-rouge">question</code> and <code class="language-plaintext highlighter-rouge">answer</code>. I created this data for my Dialogflow CX prototype, so i just converted it to a CSV format for this project. Each row in this file serves as a source for generating an individual embedding vector. Here is what the data looks like.</p>

<pre><code class="language-csv">question,answer
who is art kreimer, Art Kreimer is a seasoned Product leader ....
</code></pre>

<h3 id="the-process-flow">The Process Flow</h3>

<p>Upon receiving a user’s message, the system:</p>

<ol>
  <li>Generates an embedding for the query.</li>
  <li>Identifies the most closely aligned vectors from the stored documents.</li>
  <li>Sends the user query, along with the retrieved documents and a specific prompt, to the LLM.</li>
  <li>Present the answer to user</li>
</ol>

<p>For more information about LLM-based application architecture check out these two articles:</p>

<ul>
  <li><a href="https://eugeneyan.com/writing/llm-patterns/#retrieval-augmented-generation-to-add-knowledge">Patterns for Building LLM-based Systems &amp; Products</a></li>
  <li><a href="https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/">Emerging Architectures for LLM Applications</a></li>
</ul>

<h2 id="code-deep-dive">Code Deep Dive</h2>

<h3 id="libraries">Libraries</h3>

<p>I’m using three main tools for this chatbot:</p>

<ul>
  <li>Langchain to manage the RAG flow</li>
  <li>FAISS to store embeddings and retrieve documents. Alternative vector stores/DBs are: Chroma, Pinecone, and many others.</li>
  <li>Streamlit for UI (originally) — since migrated to Bolt for improved UX</li>
  <li>OpenAI gpt-3.5-turbo for information retrieval</li>
</ul>

<p>Here’s what the Python code looks like to bring these tools together:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">streamlit</span> <span class="k">as</span> <span class="n">st</span>
<span class="kn">from</span> <span class="nn">langchain.embeddings.openai</span> <span class="kn">import</span> <span class="n">OpenAIEmbeddings</span>
<span class="kn">from</span> <span class="nn">langchain.chat_models</span> <span class="kn">import</span> <span class="n">ChatOpenAI</span>
<span class="kn">from</span> <span class="nn">langchain.chains</span> <span class="kn">import</span> <span class="n">ConversationalRetrievalChain</span>
<span class="kn">from</span> <span class="nn">langchain.document_loaders.csv_loader</span> <span class="kn">import</span> <span class="n">CSVLoader</span>
<span class="kn">from</span> <span class="nn">langchain.vectorstores</span> <span class="kn">import</span> <span class="n">FAISS</span>
<span class="kn">from</span> <span class="nn">langchain.prompts</span> <span class="kn">import</span> <span class="n">load_prompt</span>
</code></pre></div></div>

<h3 id="prompt-engineering">Prompt Engineering</h3>

<p>Getting the chat Streamlit interface to look nice was one thing, but I spent even more time figuring out how the chatbot should talk. I tested a lot of different prompts before settling on one that works well for this task. It’s mostly accurate, but probably can be improved even more.</p>

<p>Here’s the Python code for the prompt:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">""" System: You are a CareerBot, a comprehensive, interactive resource for exploring Artiom (Art) Kreimer's background, skills, and expertise. 
Be polite and provide answers based on the provided context only. Use only the provided data and not prior knowledge.
Human: Follow exactly these 3 steps:
1. Read the context below 
2. Answer the question using only the provided Help Centre information
3. Make sure to nicely format the output so it is easy to read on a small screen.
Context : {context} 
User Question: {question}
If you don't know the answer, just say you don't know. 
Do NOT try to make up an answer.
If the question is not related to the information about Artiom Kreimer, 
politely respond that you are tuned to only answer questions about Artiom Kreimer's experience, education, training and his aspirations. 
Use as much detail as possible when responding but keep your answer to up to 200 words.
At the end ask if the user would like to have more information or what else they would like to know about Art Kreimer."""</span>
</code></pre></div></div>

<h3 id="logic">Logic</h3>

<ol>
  <li><strong>API Keys</strong>: Fetches OpenAI API keys from either the environment variables when it runs locally or Streamlit’s secrets when it runs from Streamlit cloud.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="s">"OPENAI_API_KEY"</span> <span class="ow">in</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">:</span>
    <span class="n">openai_api_key</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"OPENAI_API_KEY"</span><span class="p">)</span>

<span class="k">else</span><span class="p">:</span> <span class="n">openai_api_key</span> <span class="o">=</span> <span class="n">st</span><span class="p">.</span><span class="n">secrets</span><span class="p">[</span><span class="s">"OPENAI_API_KEY"</span><span class="p">]</span>
</code></pre></div></div>

<ol>
  <li><strong>Data and Embeddings</strong>: Loads the dataset from a CSV file and creates FAISS vectors for quick searching.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">)</span>
<span class="c1">#my_file = path+'/photo.png'
</span>
<span class="c1"># Loading prompt to query openai
</span><span class="n">prompt</span> <span class="o">=</span> <span class="n">load_prompt</span><span class="p">(</span><span class="n">path</span><span class="o">+</span><span class="s">"/templates/template1.json"</span><span class="p">)</span>
<span class="c1">#prompt = template.format(input_parameter=user_input)
</span>
<span class="c1"># loading embedings
</span><span class="n">faiss_index</span> <span class="o">=</span> <span class="n">path</span><span class="o">+</span><span class="s">"/faiss_index"</span>

<span class="c1"># Loading CSV file
</span><span class="n">data_source</span> <span class="o">=</span> <span class="n">path</span><span class="o">+</span><span class="s">"/data/about_art_chatbot_data.csv"</span>

<span class="c1"># Creating embeddings for the docs
</span><span class="k">if</span> <span class="n">data_source</span> <span class="p">:</span>
    <span class="n">loader</span> <span class="o">=</span> <span class="n">CSVLoader</span><span class="p">(</span><span class="n">file_path</span><span class="o">=</span><span class="n">data_source</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s">"utf-8"</span><span class="p">)</span>
    <span class="n">data</span> <span class="o">=</span> <span class="n">loader</span><span class="p">.</span><span class="n">load</span><span class="p">()</span>
    <span class="n">embeddings</span> <span class="o">=</span> <span class="n">OpenAIEmbeddings</span><span class="p">()</span>
    
    <span class="c1">#using FAISS as a vector DB
</span>    <span class="k">if</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">exists</span><span class="p">(</span><span class="n">faiss_index</span><span class="p">):</span>
        <span class="n">vectors</span> <span class="o">=</span> <span class="n">FAISS</span><span class="p">.</span><span class="n">load_local</span><span class="p">(</span><span class="n">faiss_index</span><span class="p">,</span> <span class="n">embeddings</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">vectors</span> <span class="o">=</span> <span class="n">FAISS</span><span class="p">.</span><span class="n">from_documents</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">embeddings</span><span class="p">)</span>
        <span class="n">vectors</span><span class="p">.</span><span class="n">save_local</span><span class="p">(</span><span class="s">"faiss_index"</span><span class="p">)</span>
    <span class="n">retriever</span><span class="o">=</span><span class="n">vectors</span><span class="p">.</span><span class="n">as_retriever</span><span class="p">()</span>
</code></pre></div></div>

<ol>
  <li><strong>Langchain Chain</strong>: Sets up a Conversational Retrieval Chain using Langchain, which combines GPT-3.5 and the FAISS vectors for responding to queries.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#Creating langchain retreval chain 
</span><span class="n">chain</span> <span class="o">=</span> <span class="n">ConversationalRetrievalChain</span><span class="p">.</span><span class="n">from_llm</span><span class="p">(</span><span class="n">llm</span> <span class="o">=</span> <span class="n">ChatOpenAI</span><span class="p">(</span><span class="n">temperature</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span><span class="n">model_name</span><span class="o">=</span><span class="s">'gpt-3.5-turbo'</span><span class="p">,</span> <span class="n">openai_api_key</span><span class="o">=</span><span class="n">openai_api_key</span><span class="p">),</span> 
                                                <span class="n">retriever</span><span class="o">=</span><span class="n">retriever</span><span class="p">,</span><span class="n">return_source_documents</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="n">verbose</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="n">chain_type</span><span class="o">=</span><span class="s">"stuff"</span><span class="p">,</span>
                                                <span class="n">max_tokens_limit</span><span class="o">=</span><span class="mi">4097</span><span class="p">,</span> <span class="n">combine_docs_chain_kwargs</span><span class="o">=</span><span class="p">{</span><span class="s">"prompt"</span><span class="p">:</span> <span class="n">prompt</span><span class="p">})</span>
</code></pre></div></div>

<ol>
  <li><strong>Chat Logic</strong>: Defines the logic for handling the conversation and maintaining chat history in Streamlit’s session state.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">conversational_chat</span><span class="p">(</span><span class="n">query</span><span class="p">):</span>
    <span class="n">result</span> <span class="o">=</span> <span class="n">chain</span><span class="p">({</span><span class="s">"system"</span><span class="p">:</span> 
    <span class="s">"You are a CareerBot, a comprehensive, interactive resource for exploring Artiom (Art) Kreimer's background, skills, and expertise. </span><span class="se">\
</span><span class="s">    Be polite and provide answers based on the provided context only. Use only the provided data and not prior knowledge."</span><span class="p">,</span> 
                    <span class="s">"question"</span><span class="p">:</span> <span class="n">query</span><span class="p">,</span> 
                    <span class="s">"chat_history"</span><span class="p">:</span> <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">[</span><span class="s">'history'</span><span class="p">]})</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">[</span><span class="s">'history'</span><span class="p">].</span><span class="n">append</span><span class="p">((</span><span class="n">query</span><span class="p">,</span> <span class="n">result</span><span class="p">[</span><span class="s">"answer"</span><span class="p">]))</span>
    
    <span class="k">if</span> <span class="s">'I am tuned to only answer questions'</span> <span class="ow">in</span> <span class="n">result</span><span class="p">[</span><span class="s">'answer'</span><span class="p">]:</span>
        <span class="k">return</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="s">"answer"</span><span class="p">])</span>
    <span class="k">else</span><span class="p">:</span> <span class="k">return</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="s">"answer"</span><span class="p">])</span>
</code></pre></div></div>

<ol>
  <li><strong>Streamlit UI for chat</strong>: Initializes the UI, displays prior messages, and manages the user input and bot responses.</li>
</ol>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#Creating Streamlit title and adding additional information about the bot
</span><span class="n">st</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">"Art Kreimer's resume bot"</span><span class="p">)</span>
<span class="k">with</span> <span class="n">st</span><span class="p">.</span><span class="n">expander</span><span class="p">(</span><span class="s">"⚠️Disclaimer"</span><span class="p">):</span>
    <span class="n">st</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">"""This is a work in progress chatbot based on a large language model.</span><span class="se">\
</span><span class="s">     It can answer questions about Art Kreimer"""</span><span class="p">)</span>
    
    <span class="k">if</span> <span class="s">"openai_model"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">:</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">[</span><span class="s">"openai_model"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"gpt-3.5-turbo"</span>

<span class="k">if</span> <span class="s">"messages"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">:</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>

<span class="k">if</span> <span class="s">'history'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">:</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">[</span><span class="s">'history'</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>

<span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">.</span><span class="n">messages</span><span class="p">:</span>
    <span class="k">with</span> <span class="n">st</span><span class="p">.</span><span class="n">chat_message</span><span class="p">(</span><span class="n">message</span><span class="p">[</span><span class="s">"role"</span><span class="p">]):</span>
        <span class="n">st</span><span class="p">.</span><span class="n">markdown</span><span class="p">(</span><span class="n">message</span><span class="p">[</span><span class="s">"content"</span><span class="p">])</span>

<span class="k">if</span> <span class="n">prompt</span> <span class="p">:</span><span class="o">=</span> <span class="n">st</span><span class="p">.</span><span class="n">chat_input</span><span class="p">(</span><span class="s">"Hi, I'm CareerBot. Ask me about Art's skills, background, or education!"</span><span class="p">):</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">.</span><span class="n">messages</span><span class="p">.</span><span class="n">append</span><span class="p">({</span><span class="s">"role"</span><span class="p">:</span> <span class="s">"user"</span><span class="p">,</span> <span class="s">"content"</span><span class="p">:</span> <span class="n">prompt</span><span class="p">})</span>
    <span class="k">with</span> <span class="n">st</span><span class="p">.</span><span class="n">chat_message</span><span class="p">(</span><span class="s">"user"</span><span class="p">):</span>
        
        <span class="n">user_input</span><span class="o">=</span><span class="n">prompt</span>
        <span class="n">st</span><span class="p">.</span><span class="n">markdown</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>

    <span class="k">with</span> <span class="n">st</span><span class="p">.</span><span class="n">chat_message</span><span class="p">(</span><span class="s">"assistant"</span><span class="p">):</span>
        <span class="n">message_placeholder</span> <span class="o">=</span> <span class="n">st</span><span class="p">.</span><span class="n">empty</span><span class="p">()</span>
        <span class="n">full_response</span> <span class="o">=</span> <span class="s">""</span>
        <span class="n">full_response</span> <span class="o">=</span> <span class="n">conversational_chat</span><span class="p">(</span><span class="n">user_input</span><span class="p">)</span>
        <span class="n">message_placeholder</span><span class="p">.</span><span class="n">markdown</span><span class="p">(</span><span class="n">full_response</span><span class="p">)</span>
    <span class="n">st</span><span class="p">.</span><span class="n">session_state</span><span class="p">.</span><span class="n">messages</span><span class="p">.</span><span class="n">append</span><span class="p">({</span><span class="s">"role"</span><span class="p">:</span> <span class="s">"assistant"</span><span class="p">,</span> <span class="s">"content"</span><span class="p">:</span> <span class="n">full_response</span><span class="p">})</span>
</code></pre></div></div>

<h2 id="conclusion">Conclusion</h2>

<p>Developing this resume bot, which leverages Streamlit, Langchain, FAISS, and the OpenAI GPT-3.5 foundational model using the RAG architecture, has been an invaluable learning experience. The RAG architecture shows significant promise; I foresee a growing number of companies incorporating this approach into their language model-based applications.</p>

<h3 id="lessons-learned">Lessons Learned</h3>

<ol>
  <li><strong>Prompt Tuning</strong>: Perfecting the prompt requires extensive trial and error to obtain accurate and context-relevant answers from the language model.</li>
  <li><strong>Data Quality</strong>: The intelligence of your chatbot is directly proportional to the quality of data you feed it.</li>
  <li><strong>Chat History</strong>: Utilizing chat history is crucial for refining the chatbot’s responses.</li>
  <li><strong>Hallucinations</strong>: Although RAG and prompt engineering have substantially reduced hallucinations, they haven’t eliminated them entirely. An additional validation step in the architecture could further mitigate this issue.</li>
  <li><strong>Data Management and Retrieval</strong>: Vector-based searches have proven effective for long and complex queries. However, this approach could benefit from adding keyword-based search methods like TF-IDF or BM25 for short queries or keyword searches. Additionally, it’s important to partition your data into manageable segments to avoid exceeding the 4K token limit imposed by the GPT-3.5 model and improve the quality of your answers.</li>
  <li><strong>Number of Chat Turns</strong>: Another consideration is to limit the number of chat turns to manage the prompt size more effectively.</li>
  <li><strong>Responsiveness</strong>: The chain I’ve developed operates at a slower pace than desired. While functional, it requires further optimization for speed.</li>
  <li><strong>Not Production-Ready</strong>: This project was an excellent learning experience, but it’s not yet ready for production deployment. Additional steps such as testing, prompt refinement, logging, monitoring, and caching for better responsiveness, as well as input and response validation, are essential to make it production-grade.</li>
</ol>

<h2 id="update-migrated-from-streamlit-to-bolt">Update: Migrated from Streamlit to Bolt</h2>

<p>Since the original post, I’ve rebuilt the frontend using <a href="https://bolt.new">Bolt</a> to improve the user experience. The core RAG architecture remains the same, but Bolt provides a cleaner, more polished interface compared to the original Streamlit app.</p>

<p>Try the live bot at <a href="https://kredar-resumegpt-imp-s3m5.bolt.host/">kredar-resumegpt-imp-s3m5.bolt.host</a> or embedded on my <a href="https://www.artkreimer.com/resume/">Resume page</a>.</p>

<h2 id="demo">Demo</h2>

<p>Check out my <a href="https://www.artkreimer.com/resume/">Resume ChatBot</a> or directly at <a href="https://kredar-resumegpt-imp-s3m5.bolt.host/">Bolt</a></p>

<div style="text-align: center;">
<iframe src="https://artkreimer.substack.com/embed" title="Newspaper sign up" height="320" width="400" style="border:1px solid #EEE; background:white;" frameborder="0" scrolling="no"></iframe>
</div>]]></content><author><name>Art Kreimer</name></author><category term="AI/ML" /><category term="NLP" /><category term="LLM" /><category term="AI" /><category term="Conversational AI" /><summary type="html"><![CDATA[Unlock the potential of RAG architecture and foundational LLM in building an advanced resume chatbot.]]></summary></entry><entry><title type="html">NLP and Text Analytics using foundational LLMs</title><link href="https://www.artkreimer.com/How-to-Analyze-App-Reviews-Using-GPT/" rel="alternate" type="text/html" title="NLP and Text Analytics using foundational LLMs" /><published>2023-06-03T00:00:00+00:00</published><updated>2023-06-03T00:00:00+00:00</updated><id>https://www.artkreimer.com/How-to-Analyze-App-Reviews-Using-GPT</id><content type="html" xml:base="https://www.artkreimer.com/How-to-Analyze-App-Reviews-Using-GPT/"><![CDATA[<h2 id="introduction">Introduction</h2>

<p>A couple of weeks ago, I completed a short course on prompt engineering - <strong>“ChatGPT Prompt Engineering For Developers”</strong> by <a href="https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/">DeepLearning.AI</a>. The best part is it’s free but only for a limited time!</p>

<p>The course provided me with valuable insights into prompt engineering techniques, which further fueled my enthusiasm for exploring the vast capabilities of Language Models. Eager to put my newfound knowledge to the test, I decided to write a script to analyze Android app reviews using the power of an LLM.</p>

<p>Gone are the days of spending countless hours and extensive data to build specific ML models for sentiment analysis, emotion detection, topic extraction, and summarization. Thanks to Large Language Models, this can now be achieved with clever prompts and several API calls. It’s absolutely mind-boggling 🤯!</p>

<p>Utilizing the techniques I learned in the course and playing with prompts,  I was able to get the sentiment, emotions, topics, summary and auto-generated response in just two API calls. Although it’s possible to achieve everything in one API call, it can get too lengthy, longer than the allowed number of tokens in one API call, I decided to split them into 2 API calls.</p>

<h2 id="prompts">Prompts</h2>

<p>The first prompt, a straightforward zero-shot learning task, requests GPT to generate the following:</p>

<ol>
  <li>One-sentence Summary: A succinct summary of each review.</li>
  <li>App Review Sentiment: Determination of whether reviews are positive, negative, or neutral.</li>
  <li>Emotion Detection: Interpretation of the emotions conveyed in the reviews.</li>
  <li>Anger and Frustration: These emotions are singled out for separate analysis.</li>
  <li>Topic Extraction: Identification of the main topics discussed in the reviews.</li>
</ol>

<p>I’ve requested GPT to deliver the response in JSON format for seamless integration into my dataframe. Despite the prompt’s simplicity, achieving optimal results required considerable experimentation and tuning. For instance, the GPT-3.5 turbo model results were inconsistent and occasionally failed to return the requested JSON format. Adding <code class="language-plaintext highlighter-rouge">Make sure the output is in a valid JSON format</code> at the end of the prompt resolved this issue. However, more rigorous testing and exception handling would be necessary for production-level applications.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">"""Your task is to perform the following actions</span><span class="se">\
</span><span class="s"> on the mobile app review delimited by triple backticks: 
1 - Summarize with 1 sentence.
2 - Determine the sentiment of the review 
3 - Identify a list of emotions in the review. Include no more than </span><span class="se">\
</span><span class="s">five items on the list. Format your answer as a list of </span><span class="se">\
</span><span class="s">lower-case words separated by commas.
4 - Identify if the writer of the app review is expressing anger.
5 - Identify if the writer of the app review is expressing frustration.
6 - Determine five topics that are being discussed in the review. </span><span class="se">\
</span><span class="s">Make each item one or two words long. Format your response as a list of items separated by commas.

Output a json object only that contains the following keys: \ 
summary, sentiment, emotions, anger as a boolean, frustration as boolean, topics.</span><span class="se">\
</span><span class="s">  Make sure the output is in a valid JSON format

Review: ```{review}```"""</span>
</code></pre></div></div>

<p>The second prompt aims to generate responses to app reviews. To achieve more consistent results, I’ve employed a few-shot learning approach. In the <code class="language-plaintext highlighter-rouge">responses_training_prompt</code>, I’ve incorporated five actual responses from the past to guide the learning process. These five instances of app reviews and corresponding responses yielded consistent and high-quality responses.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
You are part of a Customer Advocacy team that is </span><span class="se">\
</span><span class="s">responsible to respond to all customer app reviews. </span><span class="se">\
</span><span class="s">Here are some examples of reviews and responses:</span><span class="se">\
</span><span class="si">{</span><span class="n">responses_training_prompt</span><span class="si">}</span><span class="s">
Your tasks is to write a response to the mobile app review delimited by triple backticks.</span><span class="se">\
</span><span class="s">Make sure to be polite and use formal language but not too formal.</span><span class="se">\
</span><span class="s">Output a json object only that contains the following key: response. 
```name:</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s"> review:</span><span class="si">{</span><span class="n">review</span><span class="si">}</span><span class="s">```
Make sure output is a valid json format.
"""</span>
</code></pre></div></div>

<p>I recommend experimenting with prompts and parameters in the OpenAI <a href="https://platform.openai.com/playground?mode=chat">Playground</a> prior to implementing your Python code.</p>

<h2 id="python-code">Python Code</h2>

<p>Let’s delve into the Python code. We’ll first extract Google Play app reviews using the Google Play Store scraper library, then call the OpenAI API using two prompts discussed earlier to analyze Android app reviews and auto-generate responses.</p>

<p>Before we proceed, ensure you’ve created an <a href="https://auth0.openai.com/u/signup/identifier?state=hKFo2SBsNDg0R2pEUW1Ta2U4T1hxTmt2cjJsTGNZNTdDZk5USKFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIGFqVkVrR01qR3QteWlWcFphczVMSVFqZjJUMXhkaUZjo2NpZNkgRFJpdnNubTJNdTQyVDNLT3BxZHR3QjNOWXZpSFl6d0Q">OpenAI account</a> and added OpenAI API Key to your OS variables to execute the code below. For more information check <a href="https://platform.openai.com/docs/quickstart/add-your-api-key">adding an API key page</a> .</p>

<p>Let’s start by importing the necessary libraries.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">openai</span>
<span class="n">openai</span><span class="p">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"OPENAI_API_KEY"</span><span class="p">)</span>     
</code></pre></div></div>

<p>Next, we’ll retrieve Google App reviews. I’m using the <code class="language-plaintext highlighter-rouge">google_play_scraper</code> python library to fetch all reviews, but for our analysis, we’ll limit ourselves to 100 reviews. You’ll need the app package name, which can be found in the Google Play Store URL. For additional information, refer to the <a href="https://pypi.org/project/google-play-scraper/">Google-Play-Scraper documentation</a>.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># First lets use the scraper to get all Google App reviews
</span><span class="kn">from</span> <span class="nn">google_play_scraper</span> <span class="kn">import</span> <span class="n">app</span><span class="p">,</span><span class="n">Sort</span><span class="p">,</span> <span class="n">reviews_all</span>
<span class="n">all_reviews</span> <span class="o">=</span> <span class="n">reviews_all</span><span class="p">(</span>
    <span class="s">'com.wealthsimple.trade'</span><span class="p">,</span>
    <span class="n">sleep_milliseconds</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="c1"># defaults to 0
</span>    <span class="n">lang</span><span class="o">=</span><span class="s">'en'</span><span class="p">,</span> <span class="c1"># defaults to 'en'
</span>    <span class="n">country</span><span class="o">=</span><span class="s">'ca'</span><span class="p">,</span> <span class="c1"># defaults to 'us'
</span>    <span class="n">sort</span><span class="o">=</span><span class="n">Sort</span><span class="p">.</span><span class="n">NEWEST</span><span class="p">,</span> <span class="c1"># defaults to Sort.MOST_RELEVANT
</span><span class="p">)</span>
<span class="n">df_reviews</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">(</span><span class="n">all_reviews</span><span class="p">),</span><span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s">'review'</span><span class="p">])</span>
<span class="n">df_reviews</span> <span class="o">=</span> <span class="n">df_reviews</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">df_reviews</span><span class="p">.</span><span class="n">pop</span><span class="p">(</span><span class="s">'review'</span><span class="p">).</span><span class="n">tolist</span><span class="p">()))</span>
<span class="n">df_reviews</span><span class="p">[</span><span class="s">'at'</span><span class="p">]</span> <span class="o">=</span> <span class="n">df_reviews</span><span class="p">[</span><span class="s">'at'</span><span class="p">].</span><span class="n">astype</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span>
<span class="n">df_reviews</span><span class="p">.</span><span class="n">head</span><span class="p">()</span>
<span class="c1"># let's take last 100 app reviews with rating below 4 stars
</span><span class="n">last_100_reviews_df</span><span class="o">=</span><span class="n">df_reviews</span><span class="p">[</span><span class="n">df_reviews</span><span class="p">[</span><span class="s">'score'</span><span class="p">]</span><span class="o">&lt;</span><span class="mi">4</span><span class="p">][</span><span class="mi">0</span><span class="p">:</span><span class="mi">100</span><span class="p">]</span>
</code></pre></div></div>

<p>Then, we are going to add new columns to our data frame:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## I want to detect the following using openai api
</span><span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'summary'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'sentiment'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'emotions'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'anger'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'frustration'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'topics'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'review_response'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
</code></pre></div></div>

<p>We’ll establish a helper function to invoke the OpenAI API. I’ve opted for the <code class="language-plaintext highlighter-rouge">gpt-3.5-turbo</code> model. Given its speed and effectiveness, I thought it should be good for app review analysis. To enhance consistency and minimize randomness, I’ve set the <code class="language-plaintext highlighter-rouge">temperature</code> parameter to <strong>0</strong>.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_completion</span><span class="p">(</span><span class="n">prompt</span><span class="p">,</span> <span class="n">model</span><span class="o">=</span><span class="s">"gpt-3.5-turbo"</span><span class="p">):</span>
    <span class="n">messages</span> <span class="o">=</span> <span class="p">[{</span><span class="s">"role"</span><span class="p">:</span> <span class="s">"user"</span><span class="p">,</span> <span class="s">"content"</span><span class="p">:</span> <span class="n">prompt</span><span class="p">}]</span>
    <span class="n">response</span> <span class="o">=</span> <span class="n">openai</span><span class="p">.</span><span class="n">ChatCompletion</span><span class="p">.</span><span class="n">create</span><span class="p">(</span>
        <span class="n">model</span><span class="o">=</span><span class="n">model</span><span class="p">,</span>
        <span class="n">messages</span><span class="o">=</span><span class="n">messages</span><span class="p">,</span>
        <span class="n">temperature</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="c1"># this is the degree of randomness of the model's output
</span>    <span class="p">)</span>
    <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">message</span><span class="p">[</span><span class="s">"content"</span><span class="p">]</span>	
</code></pre></div></div>

<p>Next, we’ll invoke the OpenAI API using the first prompt. Given the rate limits imposed by the OpenAI API, it’s necessary to incorporate pauses between calls.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">time</span>

<span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">iterrows</span><span class="p">():</span>
    <span class="n">review</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="s">'content'</span><span class="p">]</span>
    <span class="n">prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
Your task is to perform the following actions </span><span class="se">\
</span><span class="s">on the mobile app review delimited by triple backticks: 
1 - Summarize with 1 sentence.
2 - Determine the sentiment of the review 
3 - Identify a list of emotions in the review. Include no more than </span><span class="se">\
</span><span class="s">five items on the list. Format your answer as a list of </span><span class="se">\
</span><span class="s">lower-case words separated by commas.
4 - Identify if the writer of the app review is expressing anger.
5 - Identify if the writer of the app review is expressing frustration.
6 - Determine five topics that are being discussed in the review. </span><span class="se">\
</span><span class="s">Make each item one or two words long. Format your response as a list of items separated by commas.

Output a json object only that contains the following keys: \ 
summary, sentiment, emotions, anger as a boolean, frustration as boolean, topics.</span><span class="se">\
</span><span class="s">  Make sure the output is in a valid JSON format

Review: ```</span><span class="si">{</span><span class="n">review</span><span class="si">}</span><span class="s">```
"""</span>
    
    <span class="n">response</span> <span class="o">=</span> <span class="n">get_completion</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
    <span class="k">print</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
    <span class="n">response_json</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
   
    <span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'summary'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'summary'</span><span class="p">]</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'sentiment'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'sentiment'</span><span class="p">]</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'emotions'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'emotions'</span><span class="p">]</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'anger'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'anger'</span><span class="p">]</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'frustration'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'frustration'</span><span class="p">]</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'topics'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'topics'</span><span class="p">]</span>
</code></pre></div></div>

<p>For the second part, we first need to get pairs of historical reviews and responses to construct the few-shot learning prompt:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Create a training data based on reviews and responses. We took 5 examples 
</span><span class="n">responses_training_prompt</span><span class="o">=</span><span class="s">''</span>
<span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">df_reviews</span><span class="p">[</span><span class="n">df_reviews</span><span class="p">[</span><span class="s">'repliedAt'</span><span class="p">].</span><span class="n">astype</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span><span class="o">!=</span><span class="s">'NaT'</span><span class="p">][</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">].</span><span class="n">iterrows</span><span class="p">():</span>
    <span class="n">prompt</span><span class="o">=</span><span class="sa">f</span><span class="s">"""name:</span><span class="si">{</span><span class="n">row</span><span class="p">[</span><span class="s">'userName'</span><span class="p">]</span><span class="si">}</span><span class="s"> review:</span><span class="si">{</span><span class="n">row</span><span class="p">[</span><span class="s">'content'</span><span class="p">]</span><span class="si">}</span><span class="s"> response:</span><span class="si">{</span><span class="n">row</span><span class="p">[</span><span class="s">'replyContent'</span><span class="p">]</span><span class="si">}</span><span class="s">"""</span>
    <span class="n">responses_training_prompt</span><span class="o">+=</span><span class="s">" "</span><span class="o">+</span><span class="n">prompt</span>

</code></pre></div></div>

<p>And here is the second call:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">last_100_reviews_df</span><span class="p">[</span><span class="s">'review_response'</span><span class="p">]</span><span class="o">=</span><span class="s">''</span>
<span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">iterrows</span><span class="p">():</span>
    <span class="n">review</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="s">'content'</span><span class="p">]</span>
    <span class="n">name</span><span class="o">=</span><span class="n">row</span><span class="p">[</span><span class="s">'userName'</span><span class="p">]</span>
    <span class="n">prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
You are part of a Customer Advocacy team that </span><span class="se">\
</span><span class="s">is responsible to respond to all customer app reviews. </span><span class="se">\
</span><span class="s">Here are some examples of reviews and responses:</span><span class="se">\
</span><span class="si">{</span><span class="n">responses_training_prompt</span><span class="si">}</span><span class="s">
Your task is to write a response to the mobile app review delimited by triple backticks.</span><span class="se">\
</span><span class="s">Make sure to be polite and use formal language but not too formal.</span><span class="se">\
</span><span class="s">Output a json object only that contains the following key: response. 
```name:</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s"> review:</span><span class="si">{</span><span class="n">review</span><span class="si">}</span><span class="s">```
Make sure the output is a valid JSON format.
"""</span>
    <span class="n">response</span> <span class="o">=</span> <span class="n">get_completion</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
    <span class="k">print</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
    <span class="n">response_json</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
    <span class="n">last_100_reviews_df</span><span class="p">.</span><span class="n">at</span><span class="p">[</span><span class="n">index</span><span class="p">,</span> <span class="s">'review_response'</span><span class="p">]</span><span class="o">=</span><span class="n">response_json</span><span class="p">[</span><span class="s">'response'</span><span class="p">]</span>
</code></pre></div></div>

<p>And there you have it! In roughly a hundred lines of code, we’ve determined sentiment, identified emotions, extracted topics, summarized reviews, and generated plausible responses.</p>

<h2 id="findings-and-conclusion">Findings and Conclusion</h2>

<p>Key takeaways from my exploration with LLMs: they’re akin to magic wands for NLP tasks, but they’re not without their quirks.</p>

<ol>
  <li>Crafting the right prompt is crucial. It’s no wonder Prompt Engineering is emerging as a profession. Despite completing the <a href="https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/">Deeplearning.AI course</a>, I had to experiment extensively with the phrasing to get the desired outcomes. Resources like <a href="https://www.promptingguide.ai/">PromptingGuide</a> proved beneficial. It’s about trial and error with various prompt styles and language nuances.</li>
  <li>The performance was commendable. If I were manually summarizing or extracting sentiment, the results would likely be comparable.</li>
  <li>The GPT-3.5 Turbo model I used was somewhat slow. It’s adequate for offline analysis but may lag for real-time applications. I intend to explore cheaper and faster GPT-3 models, specifically Ada.</li>
  <li>Topic extraction was a mixed bag. While it worked decently for app reviews, it was hard to identify patterns in issues. The inconsistency in topic names and extraction of obvious or irrelevant topics were notable drawbacks. I plan to refine this process using varied prompts and post-processing techniques. More updates to follow.</li>
  <li>I’ve considered comparing the cost of using OpenAI models for all these tasks against specialized ML models for specific tasks, but that’s a discussion for another day.</li>
</ol>

<p>LLMs excel in many generic NLP tasks. My next objective is to fine-tune a model and put it to the test. Stay tuned!</p>]]></content><author><name>Art Kreimer</name></author><category term="AI/ML" /><category term="NLP" /><category term="LLM" /><category term="AI" /><summary type="html"><![CDATA[Learn Large Language Models simplify sentiment analysis, emotion detection, and topic extraction, all in just two API calls.]]></summary></entry><entry><title type="html">Unlocking Better Decision-Making: Essential Mental Models for PMs</title><link href="https://www.artkreimer.com/Useful-Mental-Models-for-Product-Managers/" rel="alternate" type="text/html" title="Unlocking Better Decision-Making: Essential Mental Models for PMs" /><published>2023-03-28T00:00:00+00:00</published><updated>2023-03-28T00:00:00+00:00</updated><id>https://www.artkreimer.com/Useful-Mental-Models-for-Product-Managers</id><content type="html" xml:base="https://www.artkreimer.com/Useful-Mental-Models-for-Product-Managers/"><![CDATA[<p>Every day, we navigate a sea of information, acquire new insights, and make a myriad of minor and significant decisions. As Product Managers, we bear the crucial responsibility of making key decisions that shape the future success of our product. Effective decision-making demands an in-depth understanding of our users, market, and product, augmented by a well-curated arsenal of mental models.</p>

<p>What is a mental model? A mental model is a pattern, representation, or explanation of how something works. Our brains create patterns or models to understand complex information. We cannot keep all the details of the world in our brains, so we use models to simplify the complex into understandable and organized chunks. We use these models daily to think, make decisions, understand new concepts, and find solutions to problems. Each of us has hundreds, if not thousands, of mental models. While there are probably millions of mental models out there, not all of them are true or useful, and you don’t have to know all of them. You should know the important ones. Let’s review some of the most useful mental models for product managers.</p>

<h2 id="the-pareto-principle">The Pareto Principle</h2>

<p>The Pareto Principle, or the 80/20 rule, is a concept that states that 80% of the effects come from 20% of the causes. This principle can be applied to a variety of fields, including economics, engineering, and management. In product management, the principle can be applied in a number of ways. For example, it can be used to prioritize features for development. By identifying the top 20% of features that provide 80% of the value, product managers can ensure that their development efforts are focused on the most important features. Additionally, the principle can be used to identify areas for improvement in existing products. By analyzing which features provide the most value, product managers can identify areas where improvements can be made to increase the product’s overall value. Similarly, in the field of engineering, the principle can be used to identify the causes of defects in products. By focusing on the top 20% of the causes, engineers can eliminate 80% of the defects.</p>

<h2 id="first-principles-thinking">First Principles Thinking</h2>

<p>There are many different methods and tools to get to the solutions, but the principles are few. First principles thinking is a mental model that involves breaking down a problem into its fundamental principles and then building up from there. This approach is often used by Elon Musk, who famously said,</p>

<blockquote class="notice">
  <p>“You have to start with the fundamental truth and reason up from there.”</p>
</blockquote>

<p>People who understand the fundamental principles can use any method or come up with their own methods. However, people who blindly use methods without understanding the underlying principles are bound to make mistakes. By applying first principles thinking to product development, you can challenge assumptions and come up with innovative solutions. The 5 Whys is a problem-solving technique that involves asking “why” five times to get to the root cause of a problem. By using this technique, you can identify the underlying causes of a problem and address them directly, rather than just treating the symptoms.</p>

<h2 id="second-order-thinking">Second Order Thinking</h2>

<p>All of us can anticipate the immediate results of our actions. This is first-order thinking. Second-order thinking involves considering the subsequent effects of those actions, i.e. thinking further ahead - when making choices, considering consequences can help us avoid future problems. We must ask ourselves the critical question: “And then what?” Asking this question would help you to consider the potential long-term consequences of your choices and make more informed and strategic decisions.</p>

<p>Second-order thinking is a valuable tool to use when making decisions, prioritizing long-term interests over immediate gains and constructing effective arguments.</p>

<h2 id="occams-razor">Occam’s Razor</h2>

<p>“Occam’s Razor” is a mental model that suggests that, when presented with multiple explanations for a phenomenon, the simplest explanation is often the best one. The principle is also known as the “law of parsimony” and is commonly attributed to the medieval philosopher and theologian William of Ockham.</p>

<p>For example, if you are trying to determine why a product feature is not performing as expected, you may consider a range of explanations, such as poor user experience design, inadequate marketing, or technical issues. By applying Occam’s Razor, you would favour the explanation that requires the fewest assumptions or variables, all other things being equal. In this case, you may determine that poor user experience design is the simplest explanation, and therefore the most likely one.</p>

<p>Overall, Occam’s Razor is a useful mental model for product managers to use when evaluating options and making decisions. It encourages simplicity and clarity and can help you to prioritize and focus on the most important factors influencing a problem or decision.</p>

<h2 id="the-curse-of-knowledge">The Curse of Knowledge</h2>

<p>The Curse of Knowledge is a cognitive bias that occurs when people who are knowledgeable about a topic have difficulty explaining it to someone who is not knowledgeable. This can lead to poor communication and misunderstandings. Product managers can use this mental model to avoid jargon and technical terms when communicating with customers or other stakeholders. By using clear and simple language, you can ensure that everyone understands the message you are trying to convey.</p>

<h2 id="the-map-is-not-the-territory">The Map is not the territory</h2>

<p>Anything that is trying to model something else can’t be accurate because it’s not an actual thing. In product management, this mental model can be used to help teams recognize that different stakeholders may have different perspectives on the same issue or problem. By understanding that each person’s mental map of the situation is unique, product managers can work to find common ground and collaborate more effectively.</p>

<h2 id="conclusion">Conclusion</h2>

<p>I covered just some of the most known mental models, and as you can see, mental models are useful tools that can improve your thinking, help you learn new concepts faster and help you make better decisions in your daily life. Check out <a href="https://fs.blog/">fs.blog</a> and <a href="https://untools.co/">Untools</a> to learn more.</p>]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="Mental Models" /><category term="Product Management" /><summary type="html"><![CDATA[Get ahead in your role by adopting proven mental models. This guide unpacks the top mental frameworks that every Product Manager should know for making smarter decisions.]]></summary></entry><entry><title type="html">Best Books for Product Managers</title><link href="https://www.artkreimer.com/Must-read-Books-for-Product-Managers/" rel="alternate" type="text/html" title="Best Books for Product Managers" /><published>2022-11-05T00:00:00+00:00</published><updated>2022-11-05T00:00:00+00:00</updated><id>https://www.artkreimer.com/Must-read-Books-for-Product-Managers</id><content type="html" xml:base="https://www.artkreimer.com/Must-read-Books-for-Product-Managers/"><![CDATA[<p>I love reading books. Books are containers of knowledge and wisdom - we can all learn something from every book we read.</p>

<blockquote>
  <p>“Reading is essential for those who seek to rise above the ordinary.” - Jim Rohn<br />
“If you are going to get anywhere in life you have to read a lot of books.” - Roald Dahl<br />
“Think before you speak. Read before you think.” - Fran Lebowitz<br />
“Today a reader, tomorrow a leader.” - Margaret Fuller</p>
</blockquote>

<p>There are a lot of great books out there, and there are a lot of books on Product Management. But what are the must-reads for Product Managers?</p>

<p>I recently found this great book recommendation visual guide on <a href="https://www.delibr.com/post/visual-guide-to-the-best-books-on-product-management">Delibr</a>.</p>

<figure style="width: 50%" class="align-center">
  <img src="/assets/images/Best Product Management Books Guide.png" alt="Best Books for PMs by Delibr" />
</figure>
<!-- <figure class=""><img src="/assets/images/Best%20Product%20Management%20Books%20Guide.png"
       alt="Best Books for PMs by Delibr"><figcaption>
      Visual Guide to the best books on product management

    </figcaption></figure>
 -->

<p>Although I agree with most of these book recommendations, I would like to expand and add more books to the list. I might even update the visual at some point.</p>

<p>So here are additional must-read books for Product Managers.</p>

<h2 id="communication-and-collaboration">Communication And Collaboration</h2>

<p>Communication and collaboration are the most crucial skills for PMs at any level.</p>

<div class="feature__wrapper">

  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/never_split_the_diff.jpeg" alt="Never Split the Difference" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Never Split the Difference" by Chris Voss and Tahl Raz</h2>
          

          
            <div class="archive__item-excerpt">
              <p>As PMs, we have to constantly negotiate with stakeholders. In his book, Chris Voss, a former FBI hostage negotiator, provides insights and clear guidance everyone can apply to everyday negotiations.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3OvfsS9" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/givetake.jpeg" alt="Give and Take" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Give and Take" by Adam Grant</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Give and Take highlights what effective networking, collaboration, influence, negotiation, and leadership skills have in common.</p>

            </div>
          

          
            <p><a href="https://www.amazon.ca/Give-Take-Helping-Others-Success/dp/0143124986/ref=sr_1_1?crid=BFHTPX84UVQ2&amp;keywords=give+and+take&amp;qid=1666571162&amp;qu=eyJxc2MiOiIxLjc4IiwicXNhIjoiMS41MyIsInFzcCI6IjEuNjMifQ%3D%3D&amp;sprefix=give+nad+take%2Caps%2C111&amp;sr=8-1" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/on_writing.jpeg" alt="On Writing" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"On Writing" by Stephen King</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Product managers have to be great writers and storytellers. “On Writing” is half a personal memoir and half a writing manual - a great read from one of the masters of the craft.</p>

            </div>
          

          
            <p><a href="https://amzn.to/37MOkO5" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="uxui-design">UX/UI Design</h2>

<p>PMs are a connecting tissue between business, design and technology. They work with Product designers regularly and need a solid understanding of UX best practices.</p>

<div class="feature__wrapper">

  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/momtest.jpeg" alt="The Mom Test" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"The Mom Test" by  Rob Fitzpatrick</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Talking to your customers about their experience with your product is important. But when <a href="https://www.artkreimer.com/First-Book-2022/">everybody lies</a>, how can you get the honest answer? This book is going to show you how customer conversations can go wrong and how you can do better.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3MrMAso" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/design_norman.jpeg" alt="The Design of Everyday Things" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"The Design of Everyday Things" by Donald Norman</h2>
          

          
            <div class="archive__item-excerpt">
              <p>It is a classic in the field of design. After you read Norman’s book, you’ll notice good and bad designs everywhere. “The Design of Everyday Things” will raise your expectations about how things should be designed.</p>

            </div>
          

          
            <p><a href="https://www.amazon.ca/Design-Everyday-Things-Revised-Expanded/dp/0465050654/ref=sr_1_1?keywords=design+of+everyday+things&amp;qid=1667677691&amp;qu=eyJxc2MiOiIxLjA2IiwicXNhIjoiMC42MyIsInFzcCI6IjAuNTkifQ%3D%3D&amp;s=books&amp;sprefix=design+of+%2Cstripbooks%2C193&amp;sr=1-1" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/change_by_design.jpeg" alt="Change by Design" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Change by Design" by Tim Brown</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Tim Brown, the CEO of IDEO, introduces Design Thinking - a human-centred approach to problem-solving that helps people and organizations become more innovative and creative.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3v871n2" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="analytics-and-customer-insights">Analytics and Customer insights</h2>

<p>Great PMs can leverage qualitative and quantitative data to get deep inside into customer behaviours and their needs and leverage it to create the best possible value for their product.</p>

<div class="feature__wrapper">

  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/everybody_lies_cropped.jpeg" alt="Everybody Lies" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Everybody Lies" by Seth Stephens-Davidowitz</h2>
          

          
            <div class="archive__item-excerpt">
              <p>It’s an easy read, full of insights into human behaviour. It shows the importance of quantitative and qualitative analysis and how they complement each other.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3EasBeK" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/how_to_lie.jpeg" alt="How to Lie with Statistics" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"How to Lie with Statistics" by Darrell Huff</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Product managers need a solid foundation in statistics to be metrics-driven. This classic book is a short and fun read and will leave you smarter and more skeptical.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3EUs29r" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="strategic-thinking">Strategic Thinking</h2>

<p>To create great products, PMs need to define a great product vision and create a good strategy to achieve this vision.</p>

<div class="feature__wrapper">

  
    <div class="feature__item--left">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/good_strategy.jpeg" alt="Good Strategy, Bad Strategy" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Good Strategy, Bad Strategy" by Richard P. Rumelt</h2>
          

          
            <div class="archive__item-excerpt">
              <p>A good product/business strategy is a must. But what is a good strategy, and what are the signs of a bad strategy? Check out Rumel’s book to find answers to these questions.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3y4nnjR" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="leadership">Leadership</h2>

<p>PMs not only motivate product team members, but also provide thought leadership and mentorship to younger PMs and actively assist them in expanding and developing skills and knowledge.</p>

<div class="feature__wrapper">

  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/high_output.jpeg" alt="High Output Management" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"High Output Management" by Andy Grove</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Andy Grove was one of the greatest CEOs of our time, and his management approach is a foundation of famous OKRs (Objective Key Results). A must-read book for any manager.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3OznpGb" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/trillion_dollar.jpeg" alt="Trillion Dollar Coach" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Trillion Dollar Coach" by Eric Schmidt, Jonathan Rosenberg and Alan Eagle</h2>
          

          
            <div class="archive__item-excerpt">
              <p>Coaching is a must-have skill for any manager, and Bill Campbell was probably one of the best business coaches. He coached leaders from Google, Apple, Intuit, and many other companies. He was a great coach, community builder, and human being, and we can all learn from his legacy.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3vHjUVi" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="technical-knowledge">Technical Knowledge</h2>

<p>PMs need to have at least basic knowledge of the technical stack their product was build on and have a basic understanding on how internet works.</p>

<!-- **"Swipe to Unlock" by Neel Mehta, Aditya Agashe, Parth Detroja**  
<div style="display: block;">  
<figure style="width:20%" class="align-left">
  <!-- <a href="/assets/images/pixel_tracker_logo_120px.jpg" title="The Pixel Tracker logo" alt="The Pixel Tracker logo"> -->
<p><!-- <img src="/assets/images/swipe_to_unlock.png" alt="Swipe to unlock">
</figure>
It's a great primer on technology and business strategy. It has many references and an excellent glossary and is a must for more junior PMs.  
</div>  
[Amazon](https://amzn.to/3uBNhJf){: .btn .btn--primary}    --></p>

<div class="feature__wrapper">

  
    <div class="feature__item--left">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/swipe_to_unlock.png" alt="Swipe to unlock" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Swipe to Unlock" by Neel Mehta, Aditya Agashe, Parth Detroja</h2>
          

          
            <div class="archive__item-excerpt">
              <p>It’s a great primer on technology and business strategy. It has many references and an excellent glossary and is a must for more junior PMs.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3uBNhJf" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<h2 id="best-practices-and-culture">Best Practices and Culture</h2>
<!-- {: style="text-align: left;"}   -->

<p>There are many approaches to product management, leadership and company culture. These are some great books to expand on these topics.</p>

<div class="feature__wrapper">

  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/working_backward.jpeg" alt="Working Backwards" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Working Backwards" by Colin Bryar, Bill Carr </h2>
          

          
            <div class="archive__item-excerpt">
              <p>“Working Backwards” is a breakdown of Amazon’s approach to culture, leadership, and best practices from two long-time Amazon executives. It contains lessons and techniques you can easily apply to your company and career.</p>

            </div>
          

          
            <p><a href="https://amzn.to/3KdYiFk" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/good_to_great.jpeg" alt="Good to Great" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Good to Great" by Jim Collins</h2>
          

          
            <div class="archive__item-excerpt">
              <p>“Good to Great” is one of the first business books I read, and I really enjoyed it. It’s full of examples and tips and would be an excellent read for anyone, especially for PMs.</p>

            </div>
          

          
            <p><a href="https://amzn.to/36FkvhZ" class="btn btn--primary">Amazon</a></p>
          
        </div>
      </div>
    </div>
  
    <div class="feature__item">
      <div class="archive__item">
        
          <div class="archive__item-teaser">
            <img src="/assets/images/rework.jpeg" alt="Rework" />
            
          </div>
        

        <div class="archive__item-body">
          
            <h2 class="archive__item-title">"Rework" by Jason Fried and David Heinemeier Hansson</h2>
          

          
            <div class="archive__item-excerpt">
              <p>“Rework” is not your typical business book. It’s a quick read - a collection of small ideas/tips on Entrepreneurship, Productivity, Management, Marketing, Hiring, Business Culture, and more. The beauty of “Rework” is that it forces you to rethink what you know about productivity, management, and marketing.</p>

            </div>
          

          
            <p><a href="https://www.artkreimer.com/Rework-book-review/" class="btn btn--primary">Read More</a></p>
          
        </div>
      </div>
    </div>
  

</div>

<p>For more book recommendations check my other post <a href="https://www.artkreimer.com/Books-I-recommend/">Book Recommendations</a>.</p>

<!-- <div style="text-align: center;">
<iframe src="https://artkreimer.substack.com/embed" title="Newspaper sign up" height="320" width="400" style="border:1px solid #EEE; background:white;" frameborder="0" scrolling="no" ></iframe>
</div> -->]]></content><author><name>Art Kreimer</name></author><category term="Product Management" /><category term="Books Review" /><category term="Product Management" /><category term="Books" /><summary type="html"><![CDATA[From seasoned experts to aspiring PMs, this list of must-read books covers all the bases. Discover the reads that will sharpen your skills and mindset.]]></summary></entry></feed>