Updates, backups and monitoring

The new version is built next to the running one in /var/www/boolanga-lms-new, then the folders are swapped. The app is down for a few seconds during the swap, and the previous release stays in /var/www/boolanga-lms-prev for rollback.

1. 1

   On your computer, upload the new version.

   bashCopy

   rsync -av --exclude node_modules --exclude .next --exclude data --exclude .env \
     boolanga-lms/ you@your-server:/tmp/boolanga-lms-new/

2. 2

   On the server, put it next to the running app with the current settings.

   bashCopy

   sudo rm -rf /var/www/boolanga-lms-new
   sudo mv /tmp/boolanga-lms-new /var/www/boolanga-lms-new
   sudo cp -p /var/www/boolanga-lms/.env /var/www/boolanga-lms-new/.env
   sudo chown -R lms:lms /var/www/boolanga-lms-new

3. 3

   Back up the database.

   bashCopy

   sudo -u postgres sh -c 'pg_dump boolanga_lms | gzip > /var/backups/boolanga-lms/before-update-$(date +%F-%H%M).sql.gz'

4. 4

   Install, build and migrate while the current version keeps running.

   bashCopy

   sudo -iu lms bash -c 'cd /var/www/boolanga-lms-new && npm ci && npm run build && npm run db:migrate'

   If this fails, users aren’t affected yet. Fix the problem and run it again.

5. 5

   Swap the folders and start the new version.

   bashCopy

   sudo -iu lms pm2 delete boolanga-lms
   sudo rm -rf /var/www/boolanga-lms-prev
   sudo mv /var/www/boolanga-lms /var/www/boolanga-lms-prev
   sudo mv /var/www/boolanga-lms-new /var/www/boolanga-lms
   sudo -iu lms bash -c 'cd /var/www/boolanga-lms && pm2 start ecosystem.config.cjs && pm2 save'

6. 6

   Check it.

   sudo -iu lms pm2 logs boolanga-lms --lines 50 shows no errors, the admin panel signs in, and the rider sign-in page loads.

before-update-*.sql.gz files aren’t removed by the nightly clean-up. Delete old ones by hand.

sudo -iu lms bash -c 'cd /var/www/boolanga-lms && npm run db:refresh-completions'

Two jobs run from cron. Neither crontab contains a password.

Reminders: create the log file for lms, then edit its crontab.

sudo touch /var/log/rider-academy-reminders.log
sudo chown lms:lms /var/log/rider-academy-reminders.log
sudo crontab -u lms -e

# Reminders for unfinished training, every day at 09:00 server time
0 9 * * * cd /var/www/boolanga-lms && /usr/bin/npm run reminders >> /var/log/rider-academy-reminders.log 2>&1

Backups: create the backup folder for postgres (mode 700), then edit its crontab. postgres connects to the database without a password.

sudo install -d -o postgres -g postgres -m 700 /var/backups/boolanga-lms
sudo crontab -u postgres -e

# Nightly database backup at 02:30, kept for 14 days
30 2 * * * pg_dump boolanga_lms | gzip > /var/backups/boolanga-lms/boolanga_lms_$(date +\%F).sql.gz && find /var/backups/boolanga-lms -name 'boolanga_lms_*.sql.gz' -mtime +14 -delete

• The reminders job must cd into the app folder: the script reads .env from the current folder.
• In a crontab, % must be written as \%.
• Check the server time zone with timedatectl, and list the jobs with sudo crontab -u lms -l and sudo crontab -u postgres -l.
• A reminders run logs a line like Reminded 12 riders (0 delivered on WhatsApp). See Reminders for who gets one.

Uploaded PDFs and SCORM files aren’t stored: only the course text made from them, which is in the database.

1. 1

   Stop the app.

   bashCopy

   sudo -iu lms pm2 stop boolanga-lms

2. 2

   Recreate an empty database.

   bashCopy

   sudo -u postgres psql -c "DROP DATABASE boolanga_lms WITH (FORCE);"
   sudo -u postgres psql -c "CREATE DATABASE boolanga_lms OWNER lms;"

3. 3

   Load the backup. List the files with sudo ls /var/backups/boolanga-lms.

   bashCopy

   sudo -u postgres sh -c 'gunzip -c /var/backups/boolanga-lms/boolanga_lms_2026-09-17.sql.gz | psql -q -d boolanga_lms'

4. 4

   If the app version is newer than the backup, apply its migrations.

   bashCopy

   sudo -iu lms bash -c 'cd /var/www/boolanga-lms && npm run db:migrate'

5. 5

   Start the app and check it.

   bashCopy

   sudo -iu lms pm2 start boolanga-lms

.env must have the same PIN_SECRET (or, without it, SESSION_SECRET) as when the backup was taken, or rider PINs won’t work.

Errors worth watching for in the app log:

• Course generation failed, Translation failed: AI problems (key, rate limits, material).
• Assistant stream failed: the assistant couldn’t answer.
• SESSION_SECRET must be set: the secret is missing or shorter than 16 characters.

• PM2 restarts the app if it uses more than 1 GB of memory. Frequent restarts show in sudo -iu lms pm2 status.
• PM2 logs grow without limit. Install log rotation once: sudo -iu lms pm2 install pm2-logrotate.
• There is no health-check endpoint. The sign-in pages load even when the database is down, so for uptime monitoring call GET /api/v1/completions?limit=1 with the API key: it answers 200 only when the database works.

To list failed WhatsApp sends with their error text (the admin panel only shows “failed”):

sudo -u postgres psql -d boolanga_lms -c "select created_at, kind, error from notifications where status = 'failed' order by created_at desc limit 20;"