A few days ago, my fiancée asked me if it's possible to create an application to read the number of messages sent on WhatsApp in a given day and input this data into a spreadsheet. She was also interested in chat history from the last few days. I wasn't entirely sure how much the Meta Graph API allowed for this, but I decided to find out and help her.

And so it began.

It turned out that Meta didn't provide extensive options for fetching messages, and for real-time synchronization, only webhooks could be used (which sounded good), but supporting them required business activity confirmation and a lot of paperwork.

I decided to look for alternatives and quickly stumbled upon the library whatsapp-web.js written in Node.js, which supports real-time tracking of sent and received messages and chat history browsing. The library works based on Puppeteer and simulates a WhatsApp session from a browser.

The Application

At first glance, the architecture for the project seemed relatively straightforward, but assembling all the parts took me a while. Node.js, whatsapp-web.js, Sequelize for communication with AWS RDS (I chose a PostgreSQL database), and GCP Sheets SDK for updating the spreadsheet. I used a service account for GCP authentication and Access Key ID and Access Token for AWS authentication.

The application's entry point is:

import whatsapp from 'whatsapp-web.js';
import {syncMessage} from "../db.mjs"
import whatsappConfig from "../whatsapp.config.mjs"

const whatsappClient = new whatsapp.Client(whatsappConfig)
whatsappClient.on('message', message => syncMessage(message))
whatsappClient.on('message_create', message => syncMessage(message))

whatsappClient.initialize()

Listen for new sent or received messages and save them to the database.

K3S Deployment

At the beginning, there were several questions that I was aware of and wanted to find answers to:

• How to build a Docker image for Node.js to work headless with whatsapp-web.js?

FROM node:18

RUN apt-get update

# https://wwebjs.dev/guide/#installation-on-no-gui-systems
RUN apt install -y gconf-service libgbm-dev libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

# https://stackoverflow.com/a/25873952
RUN install -d -m 0755 -o 1000 -g 1000 /app
USER 1000
WORKDIR /app

COPY --chown=1000:1000 package.json .
COPY --chown=1000:1000 package-lock.json .
COPY --chown=1000:1000 /src ./src
COPY --chown=1000:1000 /ssl ./ssl

RUN npm install

CMD ["node", "src/cli/whatsapp.listen.mjs"]

• How can I authorize with AWS ECR to push built images?

aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin <ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com

• Which ORM to use and how?

After an initial attempt with direct interaction with pg, I decided to use the abstraction provided by the sequelize library.

• How to connect to Postgres on AWS RDS using Secrets Manager and Node.js?

db.config.mjs:

import fs from "fs"
import {configDotenv} from "dotenv";

configDotenv()

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

const secretsManagerClient = new SecretsManagerClient({
  region: process.env.AWS_DEFAULT_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
  }
})

let response;

try {
  response = await secretsManagerClient.send(
    new GetSecretValueCommand({
      SecretId: process.env.AWS_RDS_SECRET_NAME,
      VersionStage: "AWSCURRENT",
    })
  )
} catch (error) {
  throw error
}

const secret = JSON.parse(response.SecretString)

export default {
  host: process.env.AWS_RDS_HOST,
  port: 5432,
  user: secret['username'],
  password: secret['password'],
  database: process.env.AWS_RDS_DATABASE,
  ssl: {
    rejectUnauthorized: false,
    ca: fs.readFileSync("./ssl/eu-central-1-bundle.pem").toString(),
  }
}

db.mjs:

import {Sequelize} from 'sequelize'
import config from "./db.config.mjs"

const sequelize = new Sequelize(
  config.database,
  config.user,
  config.password,
  {
    dialect: "postgres",
    host: config.host,
    port: config.port,
    dialectOptions: {
      ssl: config.ssl,
    }
  }
)

• How to authorize with Google Sheets API in Node.js?

Google Cloud Platform Service Account JSON. I also had to grant access to the email from the JSON file to a spreadsheet as <service-account-name>@<project-name>.iam.gserviceaccount.com.

• How to write a k3s deployment that works on the cluster?

The initial version I used (27.08.2023) was this:

https://gist.github.com/piotr-rusin/cfa37319513c5b343227ceae59afc5dc

Several more challenges arose during work:

• How to store secrets in k3s?
• How to configure deployment to pull images from a private AWS ECR repository?

I downloaded and configured AWS CLI on the control plane:

aws configure

And added a secret of type docker-registry:

kubectl create secret docker-registry aws-ecr \
  --docker-server=https://<ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password)

And passed it in the deployment's YAML:

imagePullSecrets:
- name: "aws-ecr"

• Where and how to store Puppeteer's session and cache files?

I used hostPath:

  volumeMounts:
    - name: wwebjs-auth
      mountPath: /app/.wwebjs_auth
    - name: wwebjs-cache
      mountPath: /app/.wwebjs_cache
volumes:
  - name: wwebjs-auth
    hostPath:
      path: /home/pokemon/whataspp-metrics/.wwebjs_auth
      type: DirectoryOrCreate
  - name: wwebjs-cache
    hostPath:
      path: /home/pokemon/whataspp-metrics/.wwebjs_cache
      type: DirectoryOrCreate

After overcoming few difficulties, the deployment was successful, and the first application started on my cluster.

What's Already Done

• Dockerfile
• Bash script that builds and pushes images to AWS ECR
• K3S YAML that deploys the application on pikachu
• Real-time timestamp-saving application for WhatsApp messages to the database

Future Plans

The second part of the application remains to be written - synchronizing data from PostgreSQL into Google Sheets.

I also want to become familiar with Helm and convert the "raw" YAML deployment to a dedicated Helm chart, which I will keep in a GitHub repository. After a push, I'd like to automatically publish it to the cluster using ArgoCD.

NixOS also sounds tempting as a replacement for the current Dockerfile.