Mem0

Here’s how you can structure the code across your Next.js + TypeScript repository for the Mem0 and HumeAI WebSocket integration, along with suggested file names and their placement in the repo.

---

1. File: mem0.ts (Located in common/ai)

This file contains the Mem0 client implementation for session handling, storing preferences, conversation logs, and more. It's the core of your Mem0 integration.

common/ai/mem0.ts

import { Hume } from "hume";

// Mock implementation of Mem0Client
class Mem0Client {
  private storage: { [key: string]: any } = {};

  constructor(private apiKey: string) {}

  async createSession(userId: string): Promise<string> {
    const sessionId = `session_${userId}_${Date.now()}`;
    this.storage[sessionId] = { userId, preferences: {}, conversations: [] };
    return sessionId;
  }

  async updateSessionData(sessionId: string, data: any): Promise<void> {
    if (!this.storage[sessionId]) {
      throw new Error("Session not found");
    }
    this.storage[sessionId] = { ...this.storage[sessionId], ...data };
  }

  async getSessionData(sessionId: string): Promise<any> {
    if (!this.storage[sessionId]) {
      throw new Error("Session not found");
    }
    return this.storage[sessionId];
  }

  arrayAppend(field: string, value: any): any {
    return [...(this.storage[field] || []), value];
  }
}

const mem0Client = new Mem0Client(process.env.NEXT_PUBLIC_MEM0_API_KEY || '');

interface UserPreferences {
  preferredHotelType?: string;
  preferredActivities?: string[];
}

export interface ConversationData {
  messages: { role: 'user' | 'assistant'; content: string }[];
  timestamp: number;
}

interface Mem0Response {
  success: boolean;
  data?: any;
  error?: string;
}

async function handleMem0Operation<T>(operation: () => Promise<T>): Promise<Mem0Response> {
  try {
    const result = await operation();
    return { success: true, data: result };
  } catch (error) {
    console.error('Mem0 operation failed:', error);
    return { success: false, error: error instanceof Error ? error.message : 'Unknown error' };
  }
}

export async function initializeSession(userId: string): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionId = await mem0Client.createSession(userId);
    return sessionId;
  });
}

export async function storePreferences(sessionId: string, preferences: Partial<UserPreferences>): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionData = await mem0Client.getSessionData(sessionId);
    await mem0Client.updateSessionData(sessionId, { 
      preferences: { ...sessionData.preferences, ...preferences } 
    });
  });
}

export async function getPreferences(sessionId: string): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionData = await mem0Client.getSessionData(sessionId);
    return sessionData.preferences || {};
  });
}

export async function storeConversation(sessionId: string, conversationData: ConversationData): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionData = await mem0Client.getSessionData(sessionId);
    await mem0Client.updateSessionData(sessionId, { 
      conversations: [...sessionData.conversations, conversationData]
    });
  });
}

export async function getConversations(sessionId: string): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionData = await mem0Client.getSessionData(sessionId);
    return sessionData.conversations || [];
  });
}

export async function getConversationLog(sessionId: string): Promise<Mem0Response> {
  return handleMem0Operation(async () => {
    const sessionData = await mem0Client.getSessionData(sessionId);
    return sessionData.conversations || [];
  });
}

---

2. File: toolHandler.ts (Located in common/ai)

This handles HumeAI ToolCallMessage interactions and routes messages to the appropriate handler, such as Mem0 or any other tool you may add later.

common/ai/toolHandler.ts

import { Hume } from "hume";
import { handleMem0Message } from './mem0';  // Import Mem0 handling

export async function handleToolCallMessage(
  toolCallMessage: Hume.empathicVoice.ToolCallMessage
): Promise<Hume.empathicVoice.ToolResponseMessage | Hume.empathicVoice.ToolErrorMessage> {
  const toolHandlers: Record<string, (message: Hume.empathicVoice.ToolCallMessage) => Promise<Hume.empathicVoice.ToolResponseMessage>> = {
    mem0: handleMem0Message,
  };

  const handler = toolHandlers[toolCallMessage.name];
  
  if (handler) {
    return handler(toolCallMessage);
  }

  return {
    type: "tool_error",
    toolCallId: toolCallMessage.toolCallId,
    error: "Tool not found",
    content: "The tool you requested was not found",
  };
}

---

3. File: handleMem0Message.ts (Located in common/ai)

This file implements the logic to handle incoming Mem0-related messages and operations using the HumeAI WebSocket.

common/ai/handleMem0Message.ts

import { Hume } from "hume";
import { initializeSession, storePreferences, getPreferences, storeConversation, getConversations, getConversationLog } from './mem0';

export async function handleMem0Message(
  mem0Message: Hume.empathicVoice.ToolCallMessage
): Promise<Hume.empathicVoice.ToolResponseMessage | Hume.empathicVoice.ToolErrorMessage> {
  if (mem0Message.name === "mem0") {
    try {
      const args = JSON.parse(mem0Message.parameters) as {
        action: string;
        sessionId: string;
        data?: any;
      };

      const { action, sessionId, data } = args;

      let response;

      switch (action) {
        case "initializeSession":
          response = await initializeSession(sessionId);
          break;
        case "storePreferences":
          response = await storePreferences(sessionId, data);
          break;
        case "getPreferences":
          response = await getPreferences(sessionId);
          break;
        case "storeConversation":
          response = await storeConversation(sessionId, data);
          break;
        case "getConversations":
          response = await getConversations(sessionId);
          break;
        case "getConversationLog":
          response = await getConversationLog(sessionId);
          break;
        default:
          throw new Error("Invalid Mem0 action");
      }

      if (response.success) {
        return {
          type: "tool_response",
          toolCallId: mem0Message.toolCallId,
          content: JSON.stringify(response.data),
        };
      } else {
        throw new Error(response.error);
      }
    } catch (error) {
      return {
        type: "tool_error",
        toolCallId: mem0Message.toolCallId,
        error: "Mem0 tool error",
        content: error instanceof Error ? error.message : "Unknown error occurred",
      };
    }
  } else {
    return {
      type: "tool_error",
      toolCallId: mem0Message.toolCallId,
      error: "Tool not found",
      content: "The tool you requested was not found",
    };
  }
}

---

Suggested Repo Structure:

/app
  /components
    ChatInterface.tsx      # Frontend component for user interactions
  /common
    /ai
      mem0.ts              # Mem0 client implementation
      toolHandler.ts       # Tool call message handler
      handleMem0Message.ts # Handles Mem0-specific tool actions

How It All Fits Together:

• mem0.ts manages session data, user preferences, and conversation logs.
• toolHandler.ts is a dispatcher that routes incoming messages to the correct tool, such as Mem0.
• handleMem0Message.ts handles specific Mem0 operations like creating a session or retrieving conversation logs and sends the appropriate responses.

You can add more handlers to toolHandler.ts if you have other tools besides Mem0 in the future.

---

Thanks for the update! Given that you're using modules as components, let's integrate HumeAI and Mem0 into your existing structure with appropriate file placements and adaptations. Here's how you can structure it:

Step 1: Integrate Mem0 and HumeAI

Since you’re using HumeAI’s VoiceProvider in the Home page and the Controls module handles voice-related functionality (like muting/unmuting, connecting status, etc.), we'll integrate HumeAI and Mem0 by extending the logic in your current modules.

Folder Structure Recap

Here’s an updated breakdown of the folder structure and the modules involved:

/modules
  /Controls
    Controls.tsx         # Handles voice control UI, add HumeAI interaction here
  /MicToggle
    MicToggle.tsx        # Manages the mic on/off toggle UI
  /MicFFT
    MicFFT.tsx           # Displays visual feedback of voice input
/common
  /ai
    mem0.ts              # Mem0 client
    toolHandler.ts       # Centralized tool call handler for HumeAI
    handleMem0Message.ts # Handles Mem0-specific operations (session, preferences)

Step 2: Extend Controls.tsx for HumeAI and Mem0

Since Controls.tsx already manages voice-related interactions, this is the ideal place to integrate HumeAI and Mem0 functionality. We'll enhance this module to include onToolCall and use HumeAI’s voice-react hooks to manage WebSocket events and handle responses from Mem0.

Updated Controls.tsx

import React, { FC } from 'react'
import { useVoice } from '@humeai/voice-react'
import { Mic, MicOff, Phone } from 'lucide-react'
import { AnimatePresence, motion } from 'framer-motion'
import { Button } from '../Button'
import { cn } from '../../common/utils'
import MicFFT from '../MicFFT'
import Toggle from '../MicToggle'
import { handleToolCallMessage } from '../../common/ai'

export const Controls: FC = () => {
  const { disconnect, status, isMuted, unmute, mute, micFFT } = useVoice({
    onToolCall: handleToolCallMessage  // Handles tool calls from HumeAI (Mem0 & other tools)
  })

  return (
    <div className={cn(
      "fixed bottom-0 left-0 w-full p-4 flex items-center justify-center z-50",
      "bg-gradient-to-t from-card via-card/90 to-card/0",
    )}>
      <AnimatePresence>
        {status.value === "connected" ? (
          <motion.div
            initial={{ y: "100%", opacity: 0 }}
            animate={{ y: 0, opacity: 1 }}
            exit={{ y: "100%", opacity: 0 }}
            className="p-4 border border-border rounded shadow-sm flex items-center gap-4 bg-white"
          >
            <Toggle
              pressed={isMuted}
              onPressedChange={() => {
                if (isMuted) {
                  unmute();
                } else {
                  mute();
                }
              }}
            />
            {isMuted ? <MicOff className="size-4 border-black stroke-black" /> : <Mic className="size-4 stroke-black" />}
            <MicFFT fft={micFFT} />
            <Button onClick={disconnect}><Phone className="size-4 stroke-black" /></Button>
          </motion.div>
        ) : null}
      </AnimatePresence>
    </div>
  )
}

Step 3: Manage WebSocket Interactions with Mem0

In toolHandler.ts, we already have the function handleToolCallMessage, which routes messages to Mem0. This is used to manage actions like saving preferences, initializing sessions, etc.

Now, ensure HumeAI triggers the proper tool calls from Mem0 (as seen earlier) when interacting with voice commands:

• Store conversation logs.
• Retrieve user preferences.

Step 4: Adapt MicToggle.tsx and MicFFT.tsx

These two components handle the UI for muting/unmuting the microphone and visual feedback for voice input. These are connected to HumeAI’s voice activity through Controls.tsx and are already well-integrated.

No changes are needed here for functionality, but ensure that MicToggle can trigger voice muting/unmuting through HumeAI’s voice interaction.

Step 5: Set Up Mem0 for Handling Conversations

As discussed earlier, Mem0 manages user sessions, preferences, and conversation logs. handleMem0Message.ts will handle all Mem0-related operations based on tool call messages from HumeAI.

---

Final Setup Recap:

1. Controls.tsx: Extended to handle HumeAI tool calls (through onToolCall) for Mem0 integration.
2. MicToggle.tsx and MicFFT.tsx: No major changes; these handle the UI for voice interactions.
3. mem0.ts, toolHandler.ts, handleMem0Message.ts: Logic for Mem0 client operations and centralized message handling for HumeAI interactions.

Now your Next.js app is fully integrated with HumeAI and Mem0, allowing dynamic interactions based on voice input, with stored user data and conversation logs that are processed in real-time!

---

The StartCall.tsx module is responsible for initializing a call with HumeAI's VoiceProvider. It currently provides a button that, when clicked, starts the voice interaction if the status is not already set to "connected." Let’s modify and integrate HumeAI and Mem0 for more dynamic behavior and tool management.

1. Refine StartCall.tsx

The StartCall.tsx component will trigger a voice session, and we can enhance this to initialize a Mem0 session when the user starts the call. This would allow us to capture conversation logs or initialize user-specific preferences once the call starts.

Here’s how to integrate Mem0 session initialization with the call:

import React, { FC } from 'react'
import { useVoice } from '@humeai/voice-react'
import { Button } from '../Button'
import { initializeSession } from '../../common/ai/mem0'  // Mem0 integration

interface StartCallProps {
  onClick: () => void
}

export const StartCall: FC<StartCallProps> = ({ onClick }) => {
  const { status } = useVoice();

  const handleStartCall = async () => {
    try {
      // Initialize Mem0 session here when starting the call
      const userId = "user123"; // In real case, fetch or derive user ID dynamically
      const mem0Response = await initializeSession(userId);
      
      if (mem0Response.success) {
        console.log("Mem0 session initialized:", mem0Response.data);
        onClick(); // Proceed to start the call
      } else {
        console.error("Failed to initialize Mem0 session:", mem0Response.error);
      }
    } catch (error) {
      console.error("Error during Mem0 session initialization:", error);
    }
  };

  return (
    <Button
      className="z-50 flex items-center gap-1.5"
      onClick={handleStartCall}
      disabled={status.value === "connected"}
    >
      <span>Start Call</span>
    </Button>
  )
}

2. Explanation of Changes:

• initializeSession: When the user starts a call, we initialize a Mem0 session by calling initializeSession. This prepares Mem0 for storing any session-related information like conversation logs or user preferences.
• User ID: In a real-world application, the user ID should be derived dynamically (e.g., via authentication or session information).

3. Handle Session-Related Data during Call

Now that the session is initialized, during the voice interaction you can log conversations, update preferences, and interact with Mem0 through your handleToolCallMessage function in Controls.tsx and HumeAI’s WebSocket.

You might want to capture conversation logs like this:

import { storeConversation } from '../../common/ai/mem0';

const handleConversationLog = async (message: string) => {
  const sessionId = "session123";  // Replace with actual session ID from Mem0
  const conversationData = {
    messages: [{ role: 'user', content: message }],
    timestamp: Date.now()
  };

  const response = await storeConversation(sessionId, conversationData);
  
  if (response.success) {
    console.log("Conversation logged successfully.");
  } else {
    console.error("Failed to log conversation:", response.error);
  }
};

4. Additional Integration in the Repo

Suggested Repo Structure (including Mem0 and voice modules):

/modules
  /StartCall
    StartCall.tsx       # Updated with Mem0 session start
  /Controls
    Controls.tsx        # Handles voice commands and HumeAI interactions
  /MicFFT
    MicFFT.tsx          # Displays visual feedback of voice input
  /MicToggle
    MicToggle.tsx       # Handles the toggle UI for muting/unmuting
/common
  /ai
    mem0.ts             # Mem0 session, preference, and conversation storage logic
    toolHandler.ts      # Routes tool call messages to appropriate handlers
    handleMem0Message.ts # Handles Mem0-specific operations (session, preferences)

With this integration, when a user clicks "Start Call," a Mem0 session is created, and conversation logs and preferences can be stored during the voice interaction managed by HumeAI.

---

Yes, this Next.js API route code for interacting with the Mem0 API can definitely be useful, especially if you want to manage memories and conversations dynamically as part of your user interactions.

Let’s break down what you can do with this setup:

Key Features:

• This function sends user queries to the Mem0 API, creating new memories tied to a specific user.
• Example use case: When a user asks a question during a voice interaction, the system can store that query as a memory for future use.
• You can retrieve previously stored memories associated with a user.
• Example use case: Retrieve past conversations or logs that are important for providing personalized responses.
• Allows users to delete specific memories, which can be useful for user privacy or memory management.
• Searches across stored memories based on a query. This is useful when trying to retrieve specific information.
• The POST method handles both the creation of new memories and searching through existing memories. This could be useful for logging new interactions and searching past data for context in responses.
• The GET method retrieves all memories for a specific user.
• The DELETE method removes a memory by ID.

API Route Handler:

export const POST = async (req: NextRequest) => { /* ... */ }

Search Memories:

async function searchMemories(query: string, userId: string): Promise<string> { /* ... */ }

Delete Memories:

async function deleteMemory(memoryId: string): Promise<boolean> { /* ... */ }

Retrieve Memories:

async function getMemories(userId: string): Promise<Memory[]> { /* ... */ }

Create Memories:

async function createMemory(query: string, userId: string) { /* ... */ }

Potential Use Cases for Your Project:

• Memory Creation During Voice Interaction: You can create memories when a user interacts with the voice interface. For example, after each query in a voice session, the query can be stored using the createMemory function.
• Retrieve and Search Memories in Conversations: Use the getMemories and searchMemories functions to fetch user-specific data. This can help personalize the interaction based on past conversations or preferences.
• Memory Management UI: With the DELETE and GET routes, you can also build a UI that allows users to view, search, or delete their stored memories.

Missing Pieces/Considerations:

• Authorization and Security: You’ll need to ensure that the API routes are protected. Only the authorized user (or admin) should be able to create, view, search, or delete memories.
• Handling Large Data: Consider implementing pagination if users have many memories stored, especially when using the GET method.
• Rate Limiting: Depending on the number of API calls, rate limiting might be necessary to avoid overwhelming Mem0’s API.

How to Use in Your Project:

• Integrate with HumeAI: When a user starts a voice interaction, you could call createMemory to store the user query. Later, if the user asks a follow-up question, you could use searchMemories to find related conversations, creating a continuous conversational experience.
• Memories and Conversations in Controls: You could hook these memory functions into your Controls component or anywhere you're handling voice commands.

Next.js Setup Considerations:

Make sure you’re running Next.js 13+ with the app router for the NextRequest and NextResponse objects to work correctly in your API route.

---

In conclusion, this code provides a great backbone for managing conversations and memories dynamically with Mem0 in your app. By integrating it with your voice functionality (HumeAI), you can create a more personalized and memory-driven user experience.