https://forgejo.orwa.tech/open-source/pg_fetch_cycle

#supabase #pg_net #scraper #scrape #postgresql

Building a scraper on top of pg_net in PostgreSQL: I found myself in need of a way to chain requests together (which is what you do when you scrape). It is a simple library built on top of pg_net.

#postgresql #pg_net #scraper

PS: I also discovered you can modify the `lnav` behaviour by only showing new logs at the time of using the SSH command above:

```
#!/usr/bin/env bash

export TZ='Europe/Berlin';
NOW="$(date -Iseconds | sed -e 's/+.*$//')"
LOG_FILE="$(ls -t ~/pgdata/data/logs/*.log | head -n 1)"

if [ "$1" = "now" ]; then
lnav "$LOG_FILE" -c ":hide-lines-before $NOW"
else
lnav "$LOG_FILE"
fi
```

#ssh #lnav #postgresql #postgres #trigger #debugging #troubelshooting #pg_net #supabase

I ended up adding an SSH user called `postgreslogs` which serves as a way to SSH directly into the logs using:
- A `ForceCommand` SSH daemon config for the user
- A `sudoer.d` config to allow postgreslogs to elevate (i.e. sudo) to postgres (the DB admin account) without a password

Works really nicely! All you need to do to see the logs is to `ssh postgrelogs@postgreshost.proxmox` or similar!

#ssh #lnav #postgresql #postgres #trigger #debugging #troubelshooting #pg_net #supabase