Enhancing Zoom Call Analysis with AssemblyAI and Node.js

Integrating artificial intelligence into virtual meetings can significantly enhance efficiency and user experience. AI speech services can analyze transcriptions to generate actionable items or detailed summaries. According to AssemblyAI, this tutorial demonstrates how to obtain and parse audio from a Zoom call, transcribe the audio using AssemblyAI, and analyze it employing AssemblyAI's LeMUR API and audio intelligence models.

Setting Up Your Development Environment

To follow this tutorial, start by setting up your development environment. Install the required dependencies and securely store your AssemblyAI API key. Create a new project directory and initialize npm:

mkdir assemblyai-zoom && cd assemblyai-zoom
npm init -y
npm install assemblyai dotenv fluent-ffmpeg node-media-server

Create a .env file in your project’s root directory and add your AssemblyAI API key:

ASSEMBLYAI_API_KEY = your_api_key_here

Getting Audio from a Zoom Call

Stream the audio using Real-Time Messaging Protocol (RTMP) and save it locally as an MP3 file. Ensure you have a Pro plan or higher on Zoom and enable livestreaming for meetings.

Create a Custom Media Server

Create a mediaServer.js file in your src folder:

const NodeMediaServer = require('node-media-server');
const processAudioStream = require('./audioProcessor');

const config = {
  rtmp: {
    port: 1935,
    chunk_size: 60000,
    gop_cache: true,
    ping: 30,
    ping_timeout: 60,
  },
  http: {
    port: 8000,
    allow_origin: '*',
  },
};

const nms = new NodeMediaServer(config);

nms.on('prePublish', (id, StreamPath, args) => {
  console.log(`Stream [${id}] is about to be published at path: ${StreamPath}`);
  processAudioStream(StreamPath);
});

nms.run();

This code sets up a Node.js server for a custom streaming service.

Parse Audio from an RTMP Stream

Create a utils.js file in your src folder for unique file names:

function getFormattedDateTime() {
  const now = new Date();
  const year = now.getFullYear();
  const month = String(now.getMonth() + 1).padStart(2, '0');
  const day = String(now.getDate()).padStart(2, '0');
  const timestamp = Date.now();
  return `${year}${month}${day}_${timestamp}`;
}

module.exports = { getFormattedDateTime };

Create an audioProcessor.js file in your src folder:

const ffmpeg = require('fluent-ffmpeg');
const { getFormattedDateTime } = require('./utils');

function processAudioStream(streamPath) {
  const inputPath = `rtmp://localhost:1935${streamPath}`;
  const outputPath = `./src/recordings/meeting_${getFormattedDateTime()}.mp3`;

  ffmpeg(inputPath)
    .outputOptions('-q:a 0')
    .outputOptions('-map a')
    .on('start', (commandLine) => {
      console.log('Spawned FFmpeg with command: ' + commandLine);
    })
    .on('progress', (progress) => {
      console.log('Processing: ' + progress.timemark + '...');
    })
    .on('error', (err, stdout, stderr) => {
      console.log('An error occurred: ' + err.message);
      console.log('FFmpeg stderr: ' + stderr);
    })
    .on('end', () => {
      console.log('Processing finished!');
    })
    .save(outputPath);
}

module.exports = processAudioStream;

Start your server:

node src/mediaServer

Transcribing Audio with AssemblyAI

Initialize the AssemblyAI SDK using your API key in an assemblyai.js file:

require('dotenv').config();
const { AssemblyAI } = require('assemblyai');

const client = new AssemblyAI({
  apiKey: process.env.ASSEMBLYAI_API_KEY,
});

module.exports = client;

Transcribe a local audio file:

const client = require('./assemblyai');

const transcribeAudio = async (filePath) => {
  const transcript = await client.transcripts.transcribe({ audio: filePath });
  console.log(transcript.text);
  return transcript;
};

transcribeAudio('src/recordings/FILE_PATH_TO_AUDIO');

This code returns a full transcript of the audio file.

Analyzing Audio with LeMUR

Use AssemblyAI's LeMUR API to apply large language models (LLMs) to spoken data. For example, to summarize an audio recording:

const client = require('./assemblyai');

const summarizeAudioWithLeMUR = async (filePath) => {
  const transcript = await client.transcripts.transcribe({ audio: filePath });
  const { response } = await client.lemur.summary({
    transcript_ids: [transcript.id],
    context: 'A talk on the paradox of poverty',
    answer_format: 'bullet points',
  });
  console.log(response);
  return response;
};

summarizeAudioWithLeMUR('src/recordings/meeting_20240603_1717405934474.mp3');

This code will return a summary of the audio in bullet points.

Audio Intelligence Models

AssemblyAI provides various audio intelligence models for tasks such as content moderation, PII redaction, and sentiment analysis. For example, to redact PII from a transcript:

const transcribeAudioWithPIIRedaction = async (filePath) => {
  const transcript = await client.transcripts.transcribe({
    audio: filePath,
    redact_pii: true,
    redact_pii_policies: [
      'banking_information',
      'phone_number',
      'email_address',
    ],
    redact_pii_sub: 'hash',
  });
  console.log(transcript.text);
  return transcript;
};

transcribeAudioWithPIIRedaction('src/recordings/meeting_20240603_1717405934474.mp3');

The code will redact specified PII information, replacing it with hashes.

Conclusion

This article demonstrated how to extract audio data from a Zoom call using RTMP, Node Media Server, and FFmpeg. Additionally, it covered how to transcribe audio using AssemblyAI, apply LLMs with LeMUR, and use audio intelligence models to manipulate transcriptions.