Migrate docudjeex to Docus v4 with EN/FR content
This commit is contained in:
Djeex
@@ -0,0 +1,2 @@
|
||||
title: Bash
|
||||
icon: i-lucide-file-terminal
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
title: Servarr corrector
|
||||
description: A bash script to detect and fix duplicate media files in Sonarr and Radarr libraries by replacing copies with hardlinks to reclaim disk space.
|
||||
---
|
||||
|
||||
|
||||
:ellipsis{left=0px width=40rem top=10rem blur=140px zIndex=60}
|
||||
# Servarr duplicates corrector
|
||||
---
|
||||
|
||||
Six months after downloading terabytes of media, I realized that Sonarr and Radarr were copying them into my Plex library instead of creating hardlinks. This happens due to a counterintuitive mechanism: if you mount multiple folders in Sonarr/Radarr, it sees them as different filesystems and thus cannot create hardlinks. That’s why you should mount only one parent folder containing all child folders (like `downloads`, `movies`, `tvseries` inside a `media` parent folder).
|
||||
|
||||
So I restructured my directories, manually updated every path in Qbittorrent, Plex, and others. The last challenge was finding a way to detect existing duplicates, delete them, and automatically create hardlinks instead—to save space.
|
||||
|
||||
My directory structure:
|
||||
|
||||
```sh
|
||||
.
|
||||
└── media
|
||||
├── seedbox
|
||||
├── radarr
|
||||
│ └── tv-radarr
|
||||
├── movies
|
||||
└── tvseries
|
||||
```
|
||||
|
||||
The originals are in `seedbox` and must not be modified to keep seeding. The copies (duplicates) are in `movies` and `tvseries`. To complicate things, there are also unique originals in `movies` and `tvseries`. And within those, there can be subfolders, sub-subfolders, etc.
|
||||
|
||||
So the idea is to:
|
||||
|
||||
- list the originals in seedbox
|
||||
- list files in movies and tvseries
|
||||
- compare both lists and isolate duplicates
|
||||
- delete the duplicates
|
||||
- hardlink the originals to the deleted duplicate paths
|
||||
|
||||
Yes, I asked ChatGPT and Qwen3 (which I host on a dedicated AI machine). Naturally, they suggested tools like rfind, rdfind, dupes, rdupes, rmlint... But hashing 30TB of media would take days, so I gave up quickly.
|
||||
|
||||
In the end, I only needed to find `.mkv` files, and duplicates have the exact same name as the originals, which simplifies things a lot. A simple Bash script would do the job.
|
||||
|
||||
Spare you the endless Q&A with ChatGPT—I was disappointed. Qwen3 was much cleaner. ChatGPT kept pushing awk-based solutions, which fail on paths with spaces. With Qwen’s help and dropping awk, the results improved significantly.
|
||||
|
||||
To test, I first asked for a script that only lists and compares:
|
||||
|
||||
```sh
|
||||
#!/bin/bash
|
||||
|
||||
# Create an associative array to store duplicates
|
||||
declare -A seen
|
||||
|
||||
# Find all .mkv files only (exclude directories)
|
||||
find /media/seedbox /media/movies /media/tvseries -type f -name "*.mkv" -print0 | \
|
||||
while IFS= read -r -d '' file; do
|
||||
# Get the file's inode and name
|
||||
inode=$(stat --format="%i" "$file")
|
||||
filename=$(basename "$file")
|
||||
|
||||
# If the filename has been seen before
|
||||
if [[ -n "${seen[$filename]}" ]]; then
|
||||
# Check if the inode is different from the previous one
|
||||
if [[ "${seen[$filename]}" != "$inode" ]]; then
|
||||
# Output the duplicates with full paths
|
||||
echo "Duplicates for \"$filename\":"
|
||||
echo "${seen["$filename"]} ${seen["$filename:full_path"]}"
|
||||
echo "$inode $file"
|
||||
echo
|
||||
fi
|
||||
else
|
||||
seen[$filename]="$inode"
|
||||
seen["$filename:full_path"]="$file"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
This gave me outputs like:
|
||||
|
||||
```
|
||||
Duplicates for "episode1.mkv":
|
||||
1234567 /media/seedbox/sonarr/Serie 1/Season1/episode1.mkv
|
||||
2345678 /media/tvseries/Serie 1/Season1/episode1.mkv
|
||||
```
|
||||
|
||||
With `awk`, it would’ve stopped at `/media/seedbox/sonarr/Serie`. I’m far from an expert, but Qwen3 performed better and explained everything clearly.
|
||||
|
||||
Once I verified the output, I asked for a complete script: compare, delete duplicates, create hardlinks.
|
||||
|
||||
Again, ChatGPT disappointed. Despite my requests, it created hardlinks *before* deleting the duplicates—effectively linking and then deleting the link (though the original is kept). Not helpful.
|
||||
|
||||
Quick stopover to Qwen3, RTX 5090 in overdrive, and bam—much better result. Yes, it kept ChatGPT-style emojis, but here it is:
|
||||
|
||||
```sh
|
||||
#!/bin/bash
|
||||
|
||||
echo "🔍 Step 1: Indexing original files in /media/seedbox..."
|
||||
declare -A seen
|
||||
|
||||
# Index all .mkv files in seedbox
|
||||
while IFS= read -r -d '' file; do
|
||||
filename=$(basename "$file")
|
||||
seen["$filename"]="$file"
|
||||
done < <(find /media/seedbox -type f -name "*.mkv" -print0)
|
||||
|
||||
echo "📦 Step 2: Automatically replacing duplicates..."
|
||||
total_doublons=0
|
||||
total_ko_saved=0
|
||||
|
||||
while IFS= read -r -d '' file; do
|
||||
filename=$(basename "$file")
|
||||
original="${seen[$filename]}"
|
||||
|
||||
if [[ -n "$original" && "$original" != "$file" ]]; then
|
||||
inode_orig=$(stat -c %i "$original")
|
||||
inode_dupe=$(stat -c %i "$file")
|
||||
|
||||
if [[ "$inode_orig" != "$inode_dupe" ]]; then
|
||||
size_kb=$(du -k "$file" | cut -f1)
|
||||
echo "🔁 Replacing:"
|
||||
echo " Duplicate : $file"
|
||||
echo " Original : $original"
|
||||
echo " Size : ${size_kb} KB"
|
||||
|
||||
rm "$file" && ln "$original" "$file" && echo "✅ Hardlink created."
|
||||
|
||||
total_doublons=$((total_doublons + 1))
|
||||
total_ko_saved=$((total_ko_saved + size_kb))
|
||||
fi
|
||||
fi
|
||||
done < <(find /media/movies /media/tvseries -type f -name "*.mkv" -print0)
|
||||
|
||||
echo ""
|
||||
echo "🧾 Summary:"
|
||||
echo " 🔗 Duplicates replaced by hardlink: $total_doublons"
|
||||
echo " 💾 Approx. disk space saved: ${total_ko_saved} KB (~$((total_ko_saved / 1024)) MB)"
|
||||
echo "✅ Done."
|
||||
```
|
||||
|
||||
So, in conclusion, I:
|
||||
|
||||
- Learned many Bash subtleties
|
||||
- Learned never to blindly copy-paste a ChatGPT script without understanding and dry-running it
|
||||
- Learned that Qwen on a RTX 5090 is more coherent than ChatGPT-4o on server farms (not even mentioning “normal” ChatGPT)
|
||||
- Learned that even with 100TB of storage, monitoring it would’ve alerted me much earlier to the 12TB of duplicates lying around
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: LUKS Backup
|
||||
description: A bash script to automatically dump LUKS headers from all encrypted disks, identify them by serial number, and store them in an encrypted archive.
|
||||
---
|
||||
|
||||
|
||||
:ellipsis{left=0px width=40rem top=10rem blur=140px zIndex=60}
|
||||
# Backup of LUKS Headers for Encrypted Disks/Volumes
|
||||
---
|
||||
|
||||
I recently realized that having just the password is not enough to unlock a LUKS volume after a failure or corruption. I learned how to dump the LUKS headers from disks/volumes and to use the serial numbers along with partition names to accurately identify which header corresponds to which disk/partition (I have 10 of them!).
|
||||
|
||||
After struggling to do this manually, I asked Qwen3 (an LLM running on my RTX 5090) to create a script that automates the listing and identification of disks, dumps the headers, and stores them in an encrypted archive ready to be backed up on my backup server.
|
||||
|
||||
This script:
|
||||
|
||||
* Lists and identifies disks with their serial numbers
|
||||
* Lists partitions
|
||||
* Dumps headers into a secured folder under `/root`
|
||||
* Creates a temporary archive
|
||||
* Prompts for a password
|
||||
* Encrypts the archive with that password
|
||||
* Deletes the unencrypted archive
|
||||
|
||||
```sh
|
||||
#!/bin/bash
|
||||
|
||||
# Directory where LUKS headers will be backed up
|
||||
DEST="/root/luks-headers-backup"
|
||||
mkdir -p "$DEST"
|
||||
|
||||
echo "🔍 Searching for LUKS containers on all partitions..."
|
||||
|
||||
# Loop through all possible disk partitions (including NVMe and SATA)
|
||||
for part in /dev/sd? /dev/sd?? /dev/nvme?n?p?; do
|
||||
# Skip if the device doesn't exist
|
||||
if [ ! -b "$part" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check if the partition is a LUKS encrypted volume
|
||||
if cryptsetup isLuks "$part"; then
|
||||
# Find the parent disk device (e.g. nvme0n1p4 → nvme0n1)
|
||||
disk=$(lsblk -no pkname "$part" | head -n 1)
|
||||
full_disk="/dev/$disk"
|
||||
|
||||
# Get the serial number of the parent disk
|
||||
SERIAL=$(udevadm info --query=all --name="$full_disk" | grep ID_SERIAL= | cut -d= -f2)
|
||||
if [ -z "$SERIAL" ]; then
|
||||
SERIAL="unknown"
|
||||
fi
|
||||
|
||||
# Extract the partition name (e.g. nvme0n1p4)
|
||||
PART_NAME=$(basename "$part")
|
||||
|
||||
# Build the output filename with partition name and disk serial
|
||||
OUTPUT="$DEST/luks-header-${PART_NAME}__${SERIAL}.img"
|
||||
|
||||
echo "🔐 Backing up LUKS header of $part (Serial: $SERIAL)..."
|
||||
|
||||
# Backup the LUKS header to the output file
|
||||
cryptsetup luksHeaderBackup "$part" --header-backup-file "$OUTPUT"
|
||||
if [[ $? -eq 0 ]]; then
|
||||
echo "✅ Backup successful → $OUTPUT"
|
||||
else
|
||||
echo "❌ Backup failed for $part"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# Create a timestamped compressed tar archive of all header backups
|
||||
ARCHIVE_NAME="/root/luks-headers-$(date +%Y%m%d_%H%M%S).tar.gz"
|
||||
echo "📦 Creating archive $ARCHIVE_NAME..."
|
||||
tar -czf "$ARCHIVE_NAME" -C "$DEST" .
|
||||
|
||||
# Encrypt the archive symmetrically using GPG with AES256 cipher
|
||||
echo "🔐 Encrypting the archive with GPG..."
|
||||
gpg --symmetric --cipher-algo AES256 "$ARCHIVE_NAME"
|
||||
if [[ $? -eq 0 ]]; then
|
||||
echo "✅ Encrypted archive created: ${ARCHIVE_NAME}.gpg"
|
||||
# Remove the unencrypted archive for security
|
||||
rm -f "$ARCHIVE_NAME"
|
||||
else
|
||||
echo "❌ Encryption failed"
|
||||
fi
|
||||
```
|
||||
|
||||
**Don’t forget to back up `/etc/fstab` and `/etc/crypttab` as well!**
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
title: Socat Proxy
|
||||
description: Use socat to proxy the Docker socket through Docker Socket Proxy, allowing Beszel to collect container stats without exposing the full Docker socket.
|
||||
---
|
||||
|
||||
|
||||
:ellipsis{left=0px width=40rem top=10rem blur=140px zIndex=60}
|
||||
# Socat Proxy
|
||||
---
|
||||
|
||||
This project addresses a common use case:
|
||||
|
||||
- I have [Beszel](https://beszel.dev/), a monitoring container running in host mode, which requires access to the Docker socket to collect container statistics.
|
||||
- To avoid exposing the Docker socket fully to Beszel, I use [Docker Socket Proxy](https://github.com/Tecnativa/docker-socket-proxy), a container that sits between the Docker socket and the consuming container. It filters requests by setting appropriate permissions, preventing full exposure of the Docker socket.
|
||||
|
||||
The problem arises when **Beszel** runs in host mode. In that case, it must connect directly to **Docker Socket Proxy** on a host port, meaning the proxy’s port is exposed. This allows any container or application on the host to access it and use the Docker socket.
|
||||
|
||||
This is where [Socat Proxy](https://git.djeex.fr/Djeex/socat-proxy) comes in. It is a container that:
|
||||
|
||||
- Creates a UNIX socket
|
||||
- Listens on this socket
|
||||
- Forwards requests to Docker Socket Proxy and back
|
||||
- Replaces the real Docker socket by exposing the proxy socket in the target container via a bind mount (in this case, Beszel)
|
||||
|
||||
With this setup, Docker Socket Proxy communicates with Socat Proxy in their isolated bridge network, while the UNIX socket bind-mounted on the host has restricted permissions, preventing access from other containers or applications.
|
||||
|
||||
In short:
|
||||
|
||||
|
||||
|
||||
For example, with Beszel, the configuration would look like this:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
socat-proxy:
|
||||
image: git.djeex.fr/djeex/socat-proxy:latest
|
||||
container_name: socat-proxy-beszel
|
||||
environment:
|
||||
|
||||
- TARGET_HOST=${TARGET_HOST}
|
||||
- TARGET_PORT=${TARGET_PORT}
|
||||
- UNIX_SOCKET_PATH=${UNIX_SOCKET_PATH}
|
||||
- HOST_SOCKET_PATH=${HOST_SOCKET_PATH}
|
||||
- UNIX_SOCKET_NAME=${UNIX_SOCKET_NAME}
|
||||
volumes:
|
||||
|
||||
- ${HOST_SOCKET_PATH}:${UNIX_SOCKET_PATH}
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
|
||||
- ${TARGET_HOST}
|
||||
|
||||
socket-proxy:
|
||||
image: lscr.io/linuxserver/socket-proxy:latest
|
||||
container_name: ${TARGET_HOST}
|
||||
security_opt:
|
||||
|
||||
- no-new-privileges:true
|
||||
environment:
|
||||
|
||||
- CONTAINERS=1
|
||||
- INFO=1
|
||||
volumes:
|
||||
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
tmpfs:
|
||||
|
||||
- /run
|
||||
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent:latest
|
||||
container_name: beszel-agent
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
security_opt:
|
||||
|
||||
- no-new-privileges:true
|
||||
volumes:
|
||||
|
||||
- ${HOST_SOCKET_PATH}/${UNIX_SOCKET_NAME}:/var/run/docker.sock:ro
|
||||
environment:
|
||||
|
||||
- #... your Beszel environment variables
|
||||
depends_on:
|
||||
|
||||
- socat-proxy
|
||||
```
|
||||
|
||||
More information is available on the repository:
|
||||
|
||||
::card{title="🐋 **Socat Proxy**" to="https://git.djeex.fr/Djeex/socat-proxy" target="_blank"}
|
||||
A lightweight bind-mount socket proxy
|
||||
::
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: HotDisk
|
||||
description: A bash script that monitors hard drive temperatures and automatically shuts down the server when disks stay above a safe threshold for too long.
|
||||
---
|
||||
|
||||
|
||||
:ellipsis{left=0px width=40rem top=10rem blur=140px zIndex=60}
|
||||
# HotDisk
|
||||
---
|
||||
|
||||
When you have a NAS with several drives sitting in a laundry room, temperatures can quickly rise.
|
||||
Hard drives are very sensitive to heat and can suffer serious damage if they exceed a certain temperature threshold for too long.
|
||||
After a particularly hot summer that caused a few cold sweats while monitoring my drives’ temperatures, I started looking for a way to automatically shut down the server when disk temperatures stay above their safe limit for an extended period.
|
||||
|
||||
Since I couldn’t find a convincing solution, I decided to build my own.
|
||||
|
||||
- The script reads SMART temperature data from all SATA drives every minute.
|
||||
- It counts the number of consecutive minutes the temperature stays above or below the threshold.
|
||||
- It sends Discord notifications if the threshold is exceeded or when the temperature cools down.
|
||||
- It triggers a system shutdown if the temperature stays above the limit for the configured duration.
|
||||
- It logs all temperatures and counter states, and automatically rotates log files.
|
||||
|
||||
While I was at it, I also added an installation script that installs the main script, makes it executable, creates a systemd service and timer, and enables them automatically.
|
||||
The installer also lets you configure various parameters:
|
||||
|
||||
| Variable | Description | Default Value |
|
||||
|-----------------------|------------------------------------------------------------------------------|-----------------------------------------------|
|
||||
| `MAX_TEMP` | Maximum allowed temperature (°C) before the shutdown countdown starts | `60` |
|
||||
| `HOT_DURATION` | Consecutive minutes above `MAX_TEMP` before shutdown | `5` |
|
||||
| `COOL_RESET_DURATION` | Consecutive minutes below `MAX_TEMP` to reset all counters | `5` |
|
||||
| `LOG_FILE` | Path to the main log file | `/var/log/hdd_temp_monitor.log` |
|
||||
| `LOG_ROTATE_COUNT` | Number of log files to keep | `7` |
|
||||
| `LOG_ROTATE_PERIOD` | Log rotation period (`daily` or `weekly`) | `daily` |
|
||||
| `DISCORD_WEBHOOK` | Discord webhook URL for notifications | _Required_ |
|
||||
|
||||
It also runs another script that configures **logrotate** with the parameters defined above.
|
||||
Finally, the installer can even be executed directly via a simple `curl` command followed by one last setup script — perfect for the laziest of us.
|
||||
|
||||
I also had to handle several tricky cases: running as root without sudo, using sudo directly, running as a non-sudo user, missing dependencies, permission issues, file creation errors, disk data reading errors, and more.
|
||||
|
||||
Concurrent access to the status file also had to be managed carefully.
|
||||
|
||||
More details are available directly on the repository:
|
||||
|
||||
::card{title="📜 __HotDisk__" to="https://git.djeex.fr/Djeex/hotdisk" target="_blank"}
|
||||
Keep your drives cool!
|
||||
::
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: Backrest Docker Stop
|
||||
description: A bash script that stops Docker containers before a Backrest backup runs and restarts them after — ensuring safe database backups without complex dumps.
|
||||
---
|
||||
|
||||
|
||||
:ellipsis{left=0px width=40rem top=10rem blur=140px zIndex=60}
|
||||
# Backrest Docker Stop
|
||||
---
|
||||
|
||||
[Backrest](https://github.com/garethgeorge/backrest) is a fantastic backup tool. In the case of [Serveex](https://docu.djeex.fr/en/serveex/introduction), most of the data that needs to be backed up consists of containers, and those containers often include databases.
|
||||
|
||||
The problem? You can’t safely back up a database while it’s running. There are plenty of complex solutions involving database dumps, but often the simplest method is to stop the containers, perform the backup, and then restart them.
|
||||
|
||||
**Backrest** doesn’t natively provide this functionality, but it does allow the execution of custom scripts triggered by events, for example, at the start and end of a backup plan. Our goal is to stop the containers whose databases need to be backed up when the backup plan starts, and restart them when the backup plan finishes.To achieve this, we’ll need a small Bash script and a secure connection between Backrest and the Docker socket, to enable the following sequence:
|
||||
|
||||
- The backup plan starts
|
||||
- The event triggers the execution of a custom script
|
||||
- The script contacts Docker and retrieves a list of containers labeled `backrest.backup.stop=true`
|
||||
- It stops those containers
|
||||
- The backup plan completes
|
||||
- The event triggers another custom script
|
||||
- The script contacts Docker again, retrieves the same list, and restarts those containers
|
||||
|
||||
## Securely Connecting Backrest and Docker
|
||||
|
||||
To allow **Backrest** to communicate securely with Docker, we’ll use [Docker Socket Proxy](https://github.com/linuxserver/docker-socket-proxy).
|
||||
This avoids exposing the full Docker socket and grants only the necessary permissions.
|
||||
Here’s an example Docker stack:
|
||||
|
||||
```yaml
|
||||
---
|
||||
services:
|
||||
backrest:
|
||||
image: garethgeorge/backrest:latest
|
||||
container_name: backrest
|
||||
hostname: backrest
|
||||
security_opt:
|
||||
|
||||
- no-new-privileges:true
|
||||
volumes:
|
||||
|
||||
- ... # your volumes
|
||||
environment:
|
||||
|
||||
- ... # your environment variables
|
||||
- DOCKER_HOST=tcp://socket-proxy-backrest:2375
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
|
||||
- ... # your ports
|
||||
depends_on:
|
||||
|
||||
- socket-proxy
|
||||
|
||||
socket-proxy:
|
||||
image: lscr.io/linuxserver/socket-proxy:latest
|
||||
container_name: socket-proxy-backrest
|
||||
security_opt:
|
||||
|
||||
- no-new-privileges:true
|
||||
environment:
|
||||
|
||||
- CONTAINERS=1
|
||||
- ALLOW_START=1
|
||||
- ALLOW_STOP=1
|
||||
volumes:
|
||||
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
tmpfs:
|
||||
|
||||
- /run
|
||||
```
|
||||
|
||||
With this setup, Backrest can communicate with Docker safely and securely.
|
||||
|
||||
## The Scripts
|
||||
|
||||
Below are the scripts to use for **Backrest**’s *start* and *end* backup events.
|
||||
|
||||
::code-group
|
||||
```sh [Stop]
|
||||
#!/usr/bin/env bash
|
||||
|
||||
BACKUP_LABEL="backrest.backup.stop=true"
|
||||
BACKUP_CONTAINERS=$(docker ps -aqf "label=$BACKUP_LABEL")
|
||||
for BC in $BACKUP_CONTAINERS
|
||||
do
|
||||
docker stop "$BC"
|
||||
done
|
||||
sleep 10
|
||||
```
|
||||
|
||||
```sh [Start]
|
||||
#!/usr/bin/env bash
|
||||
|
||||
BACKUP_LABEL="backrest.backup.stop=true"
|
||||
BACKUP_CONTAINERS=$(docker ps -aqf "label=$BACKUP_LABEL")
|
||||
for BC in $BACKUP_CONTAINERS
|
||||
do
|
||||
docker start "$BC"
|
||||
done
|
||||
sleep 10
|
||||
```
|
||||
::
|
||||
|
||||
## The Label
|
||||
|
||||
Once the scripts are in place and configured for the proper **Backrest** hooks, you just need to add the label `backrest.backup.stop=true` to the `compose.yaml` files of the containers that should stop and restart during backups:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
your_service:
|
||||
...
|
||||
labels:
|
||||
|
||||
- backrest.backup.stop=true
|
||||
```
|
||||
|
||||
And that’s it!
|
||||
At the next backup, all containers with the correct label will automatically stop during the backup and restart once it’s finished.
|
||||
Reference in New Issue
Block a user