Why AI applications are actually hard? (Video)

Everyone tells you to implement 𝐑𝐀𝐆 for your AI/LLM. Everyone tells you to just load all of your data into a Vector Database, and use Langchain-type frameworks for Retrieval-Augmented Generation. But it turns out, these things are a lot easier from the Product (or feature) development perspective than from the actual Technology implementation perspective.

So I am very excited to finally have an in-depth video discussion with my partner-in-crime, Abhishek, on why technology side of AI application development is actually so hard to get right.

AI Application - an Example..

But first, consider an example. One of the very hot AI topics right now is automated SDR agent. The development process for which might look something like this:

Step 1) Product Manager starts playing with prompts on ChatGPT and comes up with a prompt prototype:

You are an expert sales development representative crafting personalized outreach messages. Your goal is to create engaging, professional messages that demonstrate value and encourage response.

Context to use:

- Company: {prospect_company_name}
- Prospect's Role: {prospect_role}
- Prospect's Name: {prospect_name}
- Company Size: {company_size}
- Industry: {industry}
- Recent News/Trigger Event: {recent_news}
- LinkedIn Activity Topics: {linkedin_topics}
- Company Website Key Points: {website_points}
- Our Product/Service: {our_solution}
- Our Unique Value Proposition: {our_uvp}
- Key Pain Points We Solve: {pain_points}
- Relevant Case Study: {case_study}

Guidelines for message generation:

1. Start with a personalized opener referencing either:
    - Recent company news or achievements
    - Prospect's LinkedIn activity
    - Industry-specific challenge
    - Mutual connection (if available)
2. Demonstrate research and relevance by:
    - Mentioning specific details about their company
    - Referencing their role and responsibilities
    - Acknowledging industry-specific challenges
3. Bridge to value proposition by:
    - Connecting their situation to a specific pain point
    - Using social proof from similar companies
    - Highlighting relevant metrics or outcomes
4. Include a clear, low-friction call-to-action that:
    - Respects their time
    - Offers specific value
    - Provides multiple response options
5. Keep the tone:
    - Professional but conversational
    - Confident but not aggressive
    - Focused on their needs, not our features
    - Brief and scannable (2-3 short paragraphs maximum)
6. Avoid:
    - Generic statements that could apply to any company
    - Technical jargon unless prospect is technical
    - Long blocks of text
    - Multiple calls-to-action
    - Excessive formatting or symbols

Please generate a message using the above context that sounds natural, not automated, and focuses on building a relationship rather than just making a sale.

Additional instructions:

- If certain context fields are empty, focus on the available information
- Adapt tone based on prospect's seniority level
- Include one specific metric or result when mentioning case studies
- Keep total message length under 150 words
- Format with appropriate spacing for readability
- End with a question that's easy to respond to

Step 2) Next stage is to map out all the systems required. This might include a waterfall approach to email address discovery, a crawler for scrapping data from LinkedIn, plus some 3rd party software for overall email deliverability and drip automation.

Step 3) Finally, all this is packaged together into some “black box” interface, and the whole thing seems like a clear winner product idea.

Except, it won’t be!

“Dumb as a rock” AI and how to avoid one

The problem with these approaches is that they blindly assume that once you have gathered some unstructured data, it is magically usable in a highly structured approach:

- Recent News/Trigger Event: {recent_news}
- LinkedIn Activity Topics: {linkedin_topics}
- Company Website Key Points: {website_points}

What they fail to recognize is that even in this simplistic example, figuring out how to work with scrapped data is a lot harder in practice than it seems.

Consider the Airtable product example I wrote about previously. Abhishek walks us through the challenges of Airtable data, and how even the use case so seemingly simple as a spreadsheet table can turn into complex spider web of decisions:

But you just need TEXT, right?

Wrong! One of the misperceptions about data preparation for LLMs is that as long as everything is in the text format, LLMs will figure out the rest.

However, when it comes to AI engineering, we don’t just care about text - we care about the meta information conveyed in text. Not all sources of text are created equal when it comes to meta data, so it is important to normalize things first:

The takeaway!

If you have watched my other podcasts before, you know there is always a single takeaway that I like the viewer to walk away with. Altogether, Abhishek and I covered about 7-10 important points one should keep in mind about data when implementing LLMs via RAG.

And it was difficult to highlight a single advice. Investors can keep calling all these things “AI wrappers”, but if this discussion has a single conclusion, it is perhaps this: building good software is still hard and you need a lot more planning when it comes to data for LLMs than just “let’s move things to a vertical database and hope for the best”.

You need a lot more planning when it comes to data management for LLMs than just “let’s move things to a vertical database and hope for the best”.