An easy way to try local AI with braindead-local-intelligence # Introduction I wanted a simple way to try local AI on my computer when I need to experiment, both a LLM to chat with (with web search and image recognition) and a model to generate or modify images. Most guides require installing a lot of things and choosing models among thousands of files, so I made a small project doing it for me. => https://git.coopix.eu/public/braindead-local-intelligence braindead-local-intelligence project page The name is a bad pun, "braindead" because you just start it to try without much thinking about it, and "local intelligence" because it runs some "intelligence" on your own computer :) In this blog post, I present the project and how to get started with it. # What it does The project runs a few containers with podman: a LLM server compatible with the OpenAI API, an image generation server, SearXNG for the web search and Open WebUI as a web interface. Everything is behind a single address, protected by a token if you need to expose it to an external service. A feature I really like is that you do not have to choose between the LLM and the image model, or to switch from one to the other. Only one model is loaded in memory at a time, the right model is loaded when a request arrives, and it is unloaded after a few minutes without any request, so your GPU memory is free again when you do not use it. There is a list of ready to use models in the documentation, chosen for regular hardware people have at home and for different needs: small text models, models understanding images, image generation models, and two models able to modify an existing image. Changing the model is a matter of copying a block of configuration and restarting, the clients do not need to change anything. => https://git.coopix.eu/public/braindead-local-intelligence/src/branch/main/docs/models.md List of the proposed models # Requirements On the software side, you only need a Linux system with podman 5.0 or newer using rootless mode (everything runs under your own user, no root involved). The project currently targets AMD graphics cards, it was never tested with NVIDIA cards because I do not have one. On the hardware side, you still need a GPU with some memory (VRAM), the models were tested up to 12 GB of VRAM, and you also need RAM because the biggest models run partly in RAM. The list should allow you to try a few things without a powerful gaming computer, although the smallest models need something like 4 GB of VRAM and 8 GB of RAM. I use a computer I built in 2021 with 32 GB of RAM and a Radeon RX 6700 XT (released in January 2021), it still works very well. # Getting started The project has its own documentaiton, but here are the main steps. Your user needs to access the GPU, and to be in the groups `render` and `video`, then log out and log in again: ``` sudo usermod -aG render,video $USER ``` From the project directory, install it and check that everything is fine, the second command should list your AMD graphics card: ``` bin/ia install ia doctor ``` The copy/paste commands will download the models in `~/.local/share/ia-locale/models`, the default ones weigh about 8 GB. Then start the services, it can take several minutes to download the containers for all runtimes: ``` ia start ``` The command `ia info` displays the address, the token and the model names to use in a client compatible with the OpenAI API. # Usage The simplest way to use it is the web interface Open WebUI at http://127.0.0.1:3000, only reachable from the computer itself. You can chat, send a picture, and enable the web search or the image generation from the `+` menu in the input box. I got good results so far with Qwen3.6 35B-A3B, coupled with the web search it does a pretty good job. It is slow though, it "thinks" before answering, and as half of the model runs in RAM, an answer can take a few minutes, but it runs locally. Having a 35B model with web search allows it to be smart enough to gather information and "understand it", without having to know everything and serve outdated information. I also use it with Camille, a Mattermost bot used within a community of friends, which replies when pinged. Everyone configures it as they want to use the model of their choice, it uses two different models depending on the request depending if it is for text or image. Mistral have a free API that allows to use ministral on it, it is a bit too dumb though, but nice if you want to give it a try. => https://github.com/jtremesay/camille/ Camille GitHub page # Conclusion Do not expect miracles, this is slow and your computer will get hot and noisy while crunching for a reply. This will always be very far from the LLMs as a service that use much more advanced models and a context window (the amount of text a model can handle at once) way bigger than what you can run at home. Even spending a few dozen of thousand of euros or dollars would not approach it. But it is a nice way to play with local AI and see what your hardware can do, and nothing leaves your computer except the web searches. As stupid as small models might sound, they could still be used for some tasks, for instance if you just need it to understand a context and say `yes` or `no`, this can be well integrated into a CI or maybe filtering local SPAM. If you can run the 35B model, it opens to a lot more of cool stuff. https://dataswamp.org/~solene/2026-11-01-braindead-local-intelligence.html #trylocal #ram #llm #way #list #gpu #user #amd #everything #ai #rss #feeds #linksharing #qwibbit

Get more memory with zram or zswap # Introduction The main laptop I use is a Lenovo x390 with only 8 GB of memory, which is not a lot nowadays with a web browser and a few applications opened. The memory cannot be upgraded on this laptop because it is soldered on it, but thanks to memory compression I can easily squeeze a few extra GB out of it. I wrote about compression with BTRFS in a previous article, it can save a lot of disk space depending on the data. This time, it is about compressing memory with zram or zswap, two features of the Linux kernel. => https://dataswamp.org/~solene/2022-09-21-btrfs-compression.html Sharing some statistics about BTRFS compression In this blog post, you will learn what zram and zswap are, how they differ, and how to check if they are doing a good job on your system. I have been experimenting with both in various cases. I found zswap to give better results in most cases, but it requires a real swap to work. # zram and zswap Both features compress memory pages, but they work in a different way. ## zram zram creates a block device stored in memory and its content is compressed. When used as a swap device, the system swaps to a part of the memory, but as the data is compressed, it takes less space than the original pages. The good point is that zram does not require any disk. The downside is that it increases the pressure on the memory, because everything relies on the memory, and when the zram device is full, there is nothing to go to unless you configured another swap. => https://docs.kernel.org/admin-guide/blockdev/zram.html Linux kernel documentation: zram ## zswap zswap requires a real swap on a storage, like a partition or a swap file. It works as a compressed cache in front of it, when a page is going to be swapped, zswap compresses it and keeps it in memory in a pool. When the pool reaches its maximum size, the oldest pages are written to the swap device. With a 3 or 4 times compression ratio, this means you can avoid swapping to the disk backed swap and get high throughput, and if non compressible data get swapped, it is sent directly to the disk. => https://docs.kernel.org/admin-guide/mm/zswap.html Linux kernel documentation: zswap ## Which one to choose? In my opinion, zswap is more efficient, unless you have an old and slow hard drive. The memory is used for what compresses well, and the swap on disk is here as a safety net when the memory is really full. zram is a good choice if you do not have a swap on disk, or if you do not want to write on it. Whatever you choose, do not use both at the same time. If zswap is enabled while zram is the swap device, zswap compresses the pages before they reach zram which compresses them again, this wastes CPU and memory for nothing. Do not cross the streams, it would be bad. Some distributions enable zram or zswap by default, so it is a good idea to check what is enabled on your system before adding the other one. # Statistics with compswap To know if memory compression is useful, you need numbers. The kernel exposes a lot of statistics in `/sys`, `/proc` and debugfs, but they are not easy to read, so I made a small tool named `compswap` to display them. => https://git.coopix.eu/public/compswap compswap git repository compswap is a shell script which detects if zram or zswap is in use and displays information like the compression algorithm, the amount of data stored, the memory really used, the compression ratio and the "effective RAM", which is the physical memory plus what is saved by compression. It also gives hints about common issues, like a double compression with zram and zswap together, a full pool or low compression ratio. Running `compswap` will display the statistics once, but you can have a regular refresh using `compswap -w 2` to refresh the output every 2 seconds. Some zswap counters are only available to root, run `sudo compswap` to display them. # Configuration There are many guides explaining how to configure zram or zswap the best way (compression algorithm, size of the pool, `vm.swappiness`), so I will not write another one. The Arch Linux wiki is a good start, even if you do not use Arch Linux. => https://wiki.archlinux.org/title/Zram Arch Linux wiki: zram => https://wiki.archlinux.org/title/Zswap Arch Linux wiki: zswap # Conclusion Memory compression is an easy way to squeeze more out of your hardware, it is free, it only costs a bit of CPU and it can give a second life to a computer with a small amount of memory. On my x390 with 8 GB, it makes a difference and I have not been able to notice any lag, more fan spinning or reduced battery life. https://dataswamp.org/~solene/2026-10-28-zram-zswap-compressed-memory.html #memory #zswap #zram #get #both #btrfs #which #statistics #rss #feeds #linksharing #qwibbit

Automated watch of vulnerabilities affecting my infrastructure # Introduction Keeping an eye on published vulnerabilities is part of my work, but there are now too many CVE published every day to read them all, and the delay between the publication of a vulnerability and its first exploitation also became very short. In my opinion, this work cannot reasonably be done by hand anymore. My infrastructure is already updated automatically for nearly all its components, so most of the time a fix arrives without me doing anything. But sometimes, I need to react faster than the update, for example to set up a workaround while waiting for a patched version. For this, I need to know quickly when a vulnerability affecting me is published. In this blog post, I share the small setup I made to receive alerts about vulnerabilities that affect my infra, with issues tracking and maybe a fix created as a PR automatically when possible. There is nothing exceptional here, but it may give ideas to some readers. # How it works My infrastructure is described entirely as code in a git repository hosted on my Forgejo instance. Forgejo is a self-hosted "forge", like Gitlab, and it comes with a CI compatible with GitHub Actions. This CI job runs three times a day, in the morning before I wake up, at noon before I go back to work, and in the evening just before the end of my work day, in case it is useful for my job. It does the following: * build an inventory of what runs in my infrastructure from the repository (container images and system packages) * fetch the recent announcements from a list of security feeds * send the inventory and the announcements to a language model, which keeps only the serious vulnerabilities affecting my setup * for each new alert, ask the model to create a fix as a pull request if possible * open an issue in Forgejo to keep track of the alert * warn me over XMPP The model only sees a high level view of the infrastructure, which is enough to know which software runs where and how it is updated. The prompt asks it to ignore what does not matter to me (denial of service only, XSS, products I do not use) and also to suggest a workaround, like the commands to run or a configuration change such as disabling the vulnerable feature or blocking a path in the reverse proxy. The issue title contains the CVE identifier that is checked before opening a new issue to avoid duplicates, and I close the issue when it is handled or if it is a false positive. The pull requests are never merged automatically, they are only here to save me time, especially when the fix is trivial like changing the version of a container image. Of course, they must be reviewed before merging. # Sources Here are some of the feeds I use: * the CISA KEV catalog, the list of vulnerabilities known to be actively exploited * the GitHub security advisories with critical and high severity, most of the projects I am using are published on GitHub so it works fine * the Ubuntu security notices, for the packages of my Ubuntu containers * the oss-security mailing list, where open source projects often announce issues before other sources => https://www.cisa.gov/known-exploited-vulnerabilities-catalog CISA Known Exploited Vulnerabilities Catalog => https://github.com/advisories GitHub Advisory Database => https://ubuntu.com/security/notices Ubuntu Security Notices => https://www.openwall.com/lists/oss-security/ oss-security mailing list # The language model I use the Mistral API with the model `ministral-14b`, it is available with their free offer. From my experience, a 14B model (14 billions parameters) is good enough for this task, because it does not have to be smart, it only needs to read announcements and compare them with my inventory. I chose Mistral because I need to improve my skills about AI, it is a provider I had never used before, and it is the only one I found with a free offer giving access to an API. This would work with any other provider without problem. A 14B model also runs on hardware that an individual can have at home for a reasonable price, so this could run on-premise. # Notifications I chose XMPP for the alerts because I have an XMPP client on all my systems and the notifications work well everywhere. The message is sent with `go-sendxmpp` from a dedicated account. This is really a personal choice, it could have been Matrix, SMS, a Mattermost, Slack or Teams webhook, whatever floats your boat. => https://salsa.debian.org/mdosch/go-sendxmpp go-sendxmpp project page For each alert, I get two things: * an issue in Forgejo explaining why the vulnerability was classified as relevant, with its severity and a link to the source * an XMPP message with a link to the issue, a short summary and the affected components # Conclusion It has not been running for long, so it is still being tuned, but it already caught CVE-2026-111497 quickly, a serious remote code execution vulnerability. This setup does not replace my usual cybersecurity watch, I still read security advisories and releases feeds. It completes it, especially on days where I am really busy and that I could have missed something important. https://dataswamp.org/~solene/2026-10-23-automated-vulnerability-watch.html #infrastructure #vulnerabilitiesaffecting #introductionkeeping #cve #rss #feeds #linksharing #qwibbit

Expose only share links of a private service with caddy # Introduction In my self-hosted infrastructure, most services are only reachable through my VPN. But some of them have a feature to share something with people outside: Immich has share links for albums and photos, Nextcloud has share links for files and folders. => https://dataswamp.org/~solene/2026-10-15-selfhosting-infrastructure-incus-os.html My self-hosted infrastructure with Incus OS I wanted to add some extra security, but the people receiving a link should not have to install a VPN client or create an account, they must be able to click on it and see the content. The usual way to do this is to expose the whole service on the Internet, and then you have to trust the login page, the API and every route of the application. I found a simple pattern with the reverse proxy caddy that only lets in requests carrying a valid share link, and answers a 404 to everything else. The service itself is used to check if the link is valid. => https://caddyserver.com/ caddy official website In this blog post, you will learn how to use caddy to expose only the "share link" part of a service, how it works, and what it brings for security. # The idea: forward authentication caddy has a directive named `forward_auth`. Before forwarding a request to the service, caddy sends a first request to another server (the "auth server"), and looks at the answer. If the answer is a success, the real request is forwarded, otherwise caddy returns the answer of the auth server to the client. This is usually used with software like Keycloak + oauth-proxy, to add a login page in front of an application. => https://caddyserver.com/docs/caddyfile/directives/forward_auth caddy documentation: forward_auth directive Here, the auth server is the service itself. Most applications have a route answering differently if a share key is valid or not, so we can ask the application "is this link valid?" before letting the request in. The `forward_auth` directive is actually a shortcut for a `reverse_proxy` with a few options. I did not use `forward_auth` directly because I needed to rewrite the URL of the check request and to accept another status code than 2xx, so I wrote the long form with `reverse_proxy` and `handle_response`. This is also easier to understand, because everything is visible in the configuration. => https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#intercepting-responses caddy documentation: reverse_proxy, intercepting responses The whole design is: * the full service is reachable only through the VPN (or with an IP restriction), on a separate caddy instance (a single caddy server could work, but be sure to put IP restrictions on the internal vhost) * a public caddy site exposes only the routes needed by a guest, matched very strictly * for each request, caddy asks the service if the share key in the URL is valid * if it is valid, the request is forwarded, if not, caddy answers 404 * every other URL gets the same 404, so a visitor cannot even know which software is behind # Example with Immich share links Immich is a photo and video library, I use it for my photos and videos. Browsing the library and uploading only works through the VPN, but I want to share albums with people who do not have direct access to it. ## The check A share link in Immich looks like `https://share.example.com/share/`, and then the web interface calls the API with `?key=` in the URL. I needed an API route that tells if a key is valid without doing anything. I found that requesting `/api/users/me?key=` gives: * 403 if the key is valid, because a share link is not allowed to read the user profile * 401 if the key is invalid or expired So here, a 403 means the key is valid. The password of a share link (if there is one) does not matter for this check, Immich still asks for it after. This is a side effect of how Immich checks permissions, not a feature made for this purpose, so a future version of Immich could change it. If this ever changes, the share links will answer 404 and I will notice quickly. In the caddy configuration, I wrote a snippet (a reusable block of configuration, called with `import`) taking the key as argument: ``` (share_key_immich) { reverse_proxy http://10.0.0.10:2283 { method GET rewrite /api/users/me?key={args[0]} @valid_key status 403 handle_response @valid_key { vars share_key valid } handle_response { respond 404 } } reverse_proxy http://10.0.0.10:2283 } ``` The first `reverse_proxy` is the check: * `method GET` and `rewrite` replace the method and the URL of the request sent to Immich, so whatever the visitor asked, Immich only receives `GET /api/users/me?key=...` * `@valid_key status 403` matches the answer of Immich when the key is valid * when the answer matches, the `handle_response` block does not write any response (the `vars` line only sets an unused variable), so caddy continues with the next directive, which is the second `reverse_proxy` forwarding the real request * in any other case, caddy answers 404 and the request never reaches Immich ## The public site Then, the public site only accepts the URLs used by a share page. Everything is matched with strict regular expressions, so a guest can only reach a few API routes, and only with a key that looks like a key. ``` share.example.com { # a visitor must not be able to send its own credentials request_header -X-Immich-Share-Key request_header -X-Immich-Share-Slug request_header -X-Immich-User-Token request_header -X-Immich-Session-Token request_header -X-Api-Key request_header -Authorization # page of a share: /share/ and /share//photos/ @share_page path_regexp page ^/share/([A-Za-z0-9_-]+)(/photos/[0-9a-f-]{36})?/?$ # API routes used by a share page, with ?key= @api_read { method GET path_regexp ^/api/(shared-links/me|albums/[0-9a-f-]{36}(/map-markers)?|assets/[0-9a-f-]{36}(/original|/thumbnail|/video/playback|/video/stream/.+)?|timeline/buckets?)$ expression `{query.key}.matches("^[A-Za-z0-9_-]+$")` } @api_post { method POST path /api/shared-links/login /api/download/info /api/download/archive expression `{query.key}.matches("^[A-Za-z0-9_-]+$")` } # requests without a key, and static files of the web interface @share_config { method GET path /api/server/config /api/server/features header Referer https://share.example.com/share/* } @share_static { method GET path /_app/* header Referer https://share.example.com/* } handle @share_page { import share_key_immich {re.page.1} } handle @api_read { import share_key_immich {query.key} } handle @api_post { import share_key_immich {query.key} } handle @share_config { reverse_proxy http://10.0.0.10:2283 } handle @share_static { reverse_proxy http://10.0.0.10:2283 } handle { respond 404 } } ``` The `request_header -...` lines remove all the headers Immich uses for authentication, this way the share key in the URL is the only way to be authenticated on this site. The key is taken from the path for the share page (`{re.page.1}` is the first group of the regular expression), and from the `key` parameter of the query for the API calls (`{query.key}`). The server configuration and the JavaScript and CSS files of the interface do not carry any key, so I only allow them with a `Referer` header coming from the site. A client can put anything in this header, it is not a security measure, it just avoids answering to random scanners. These routes only give the static files of the interface and some general server information, which is ok to expose. To find which routes a share page needs, the easiest way is to open a share link with the developer tools of your browser, look at the requests in the "Network" tab, and allow them one by one until the page works. This is not ideal, but it works. # Example with Nextcloud share links I am running a Nextcloud instance through the VPN only, but I want others to be able to open the files and folders I occasionally share. ## The check A Nextcloud share link looks like `https://cloud.example.com/s/`, where the token is a random string identifying the share. This time there is no need for a strange API trick, the password page of a share `/s//authenticate/showShare` answers 200 if the token is valid and 404 if it is not. It does not ask for the password, and it does not display the share, so this is a good route for the check and only a 200 lets the request in. If the share is protected by a password, this is not a problem because the check only tells if the token exists, Nextcloud will still ask for the password. This is the same pattern, so the snippet is almost the same as for Immich: ``` (share_token_nextcloud) { reverse_proxy http://10.0.0.30:19000 { method GET rewrite /s/{args[0]}/authenticate/showShare @valid_token status 200 handle_response @valid_token { vars share_token valid } handle_response { respond 404 } } reverse_proxy http://10.0.0.30:19000 } ``` ## The public site The public site allows the share page, the download and preview of shared files, the password page, and the public WebDAV of the share. WebDAV is used by Nextcloud to list the files of a shared folder, and to upload files into it. ``` cloud.example.com { request_header -Authorization # page of a share, download, direct preview @share_page { method GET path_regexp page ^(?:/index\.php)?/s/([A-Za-z0-9-]+)(/download(/[^/]+)?|/preview)?/?$ } # page and form of the password of a protected share @share_auth { method GET POST path_regexp auth ^(?:/index\.php)?/s/([A-Za-z0-9-]+)/authenticate/[A-Za-z]+$ } @share_preview { method GET path_regexp preview ^(?:/index\.php)?/apps/files_sharing/publicpreview/([A-Za-z0-9-]+)$ } # public WebDAV of the share: listing, download, upload (simple or in chunks) # Nextcloud applies the permissions of the link @share_dav { method GET HEAD PROPFIND PUT MKCOL MOVE COPY DELETE path_regexp dav ^/public\.php/dav/(?:files|uploads)/([A-Za-z0-9-]+)(/.*)?$ } # requests without a token and static files of the interface (including the PDF viewer), from a page of the site # I found this regex in a configuration example to serve static data separately @static { method GET path_regexp ^(?:/(?:core|dist|apps/[a-z0-9_]+)/.+\.(?:js|mjs|css|map|svg|png|gif|jpg|webp|ico|woff2?|ttf|json|ftl|bcmap|icc|pfb|wasm)|(?:/index\.php)?/apps/files_pdfviewer/|(?:/index\.php)?/apps/theming/(?:favicon|icon|image|img|manifest|theme|css)(?:/.*)?|(?:/index\.php)?/(?:csrftoken|core/mimeicon)|/index\.php/apps/files/preview-service-worker\.js)$ header Referer https://cloud.example.com/* } handle @share_page { import share_token_nextcloud {re.page.1} } handle @share_auth { import share_token_nextcloud {re.auth.1} } handle @share_preview { import share_token_nextcloud {re.preview.1} } handle @share_dav { import share_token_nextcloud {re.dav.1} } handle @static { reverse_proxy http://10.0.0.30:19000 } handle { respond 404 } } ``` Nextcloud URLs can start with `/index.php` or not, this is why the regular expressions have an optional `(?:/index\.php)?` at the beginning. The WebDAV route allows methods that write or delete files and Nextcloud only accepts them if the share link allows it, for example a read only share still refuses an upload. You can not replace Nextcloud authorization with caddy here. The long list of extensions for static files is needed by the PDF viewer, and the `Referer` check is the same as for Immich. All requests will answer 404, except if they target a known working share URL or that a request has a working URL in its Referer (this is the trick to have other resources part of the request to be downloaded). # What it brings for security An attacker scanning the Internet only gets 404 replies. The software behind is not visible, so bots looking for a specific application will not find it. If there is a vulnerability in the login page, the admin API or the upload feature, it cannot be reached from the Internet, only the code needed to display a share is exposed, and only to someone who already has a valid link. This is not entirely security by obscurity here, the main goal is not to hide Nextcloud or Immich, but prevent anyone to trigger the backend code if they do not have a legitimate need. A recent example is php2shell, the name given to two critical flaws in PHP itself (CVE-2026-109618 and CVE-2026-109619), which could allow remote code execution without authentication. At the time of writing, it is still not clear which part of PHP is vulnerable. Nextcloud is written in PHP, so a public Nextcloud is a good target. With this setup, almost every request without a valid share link gets a 404 from caddy and never reaches PHP, it is much harder to even send the exploit. A few routes without a token still go through PHP (like `/csrftoken` or the theming icons), so it does not fully protect against this kind of flaw. This only works if the public site stays strict. If you also allow the normal login page of the service, the protection does not bring much anymore, because the login page is not a secret, anyone can reach it, and with it most of the application code and its vulnerabilities. Same thing for an API route allowed "just in case". On my public sites, a route either checks a share token with the service, or gives harmless content like the static files of the interface, everything else stays on the VPN. Guessing a link is not realistic, share keys in Immich and Nextcloud are long and random, and an invalid key gives the same 404 as any other URL. For the people I share things with, nothing changes. They receive a link and it opens in their browser, they do not need to install anything or create an account. On my side, I keep using the full services through the VPN, with a dedicated domain name. # Limitations Someone with a valid link can still reach the routes allowed on the public site, so a vulnerability in these routes would still be exploitable, but by far fewer people. The routes depend on how the application works. An update may add or rename a route, and the share page will break until I update the configuration. There are also smaller things: * each allowed request triggers a second request to the service for the check * if someone forwards a share link, the new person has access too, but this is how share links work anyway Finding which routes are needed takes some time at first, the browser developer tools help a lot for this. Writing the regular expressions and checking that the links still work correctly is a bit time consuming. # Conclusion This setup is simple once you have the few lines of caddy configuration per service, and it does the job, the services are not visible on the Internet anymore, except for the people I explicitly gave a link to. So far, I am happy with this implementation. I originally had a simpler version filtering unused routes, but it still gave too much access to the actual API answering content. This pattern can be used for any service that has a way to tell if a share link or a token is valid. If you self-host a service only for a small set of users, but sometimes need to share something with someone outside, it is worth a try. # Going further If you want a real login page in front of a service, you can use `forward_auth` with an authentication server like Keycloak with oauth-proxy, this is what this directive was made for, but this is a different need than sharing a link with someone. https://dataswamp.org/~solene/2026-10-21-caddy-forward-auth-public-share-links.html #rss #feeds #linksharing #qwibbit

My self-hosted infrastructure with Incus OS # Introduction For the last seven months, I spent a good part of my (small amount of) free time rebuilding my self-hosting infrastructure at home. I wanted something secure, fully automated, that updates itself and that I can rebuild from scratch if something goes wrong. Most of the ideas come from my work on infrastructures following SecNumCloud (a French security standard for cloud providers) and, to a lesser extent, ISO 27001, I wanted to see how far I could apply the same principles at home. In this blog post, I will explain how my personal infrastructure works, how all the pieces fit together, and why I think each security layer is useful. This is not a step by step guide, I will not go deep into code, the goal is to show the architecture and the reasons behind it. Before going further, I need to be honest, this is a personal infrastructure running on a single machine at home, and it is probably totally overengineered for most people. If you just want to host a Nextcloud for your family, a lot of what follows is not needed. But I learned a lot along the way, especially about containers and cloud-init, and I am very happy with the result :) The hosting server is nothing special, just a regular laptop with 16 GB of RAM, a NVMe SSD and a network card, sitting next to my Internet provider box. # The big picture Everything runs on this laptop. The operating system is Incus OS, a small immutable system that only run Incus, a manager for system containers and virtual machines (it is like Proxmox but more container oriented). => https://linuxcontainers.org/incus-os/ Incus OS official website On top of it, there are about twenty Ubuntu system containers, one per service, and two virtual machines for the more sensitive parts. Inside each system container, the applications themselves run as podman containers, described with "quadlets", which are files that systemd turns into services. So there are two levels of containers: Incus instances, and podman containers inside them. I am currently hosting the following services: * Nextcloud * Forgejo, with a CI runner * Immich for photos/videos dumped from smartphones * PeerTube for videos * Vaultwarden for passwords * Kanboard * Uptime Kuma and SmokePing to monitor the network * a static website * a monitoring stack: Prometheus, Grafana, Loki, Alertmanager * CrowdSec * restic for backups Nothing is configured by hand. Everything is declared from a git repository with OpenTofu (the open source fork of Terraform) and cloud-init (the tool that configures a cloud system at its first boot). The system containers are unprivileged, which means the root user inside a container is not root on the host. Compared to a VM per service, this is very efficient: the RAM and disk usage are much lower, and everything is pretty fast. Here is a diagram of how everything fits together: => static/selfhosting-infrastructure-incus-os.svg Diagram of the infrastructure # Incus OS Incus OS is the foundation, I chose it because it is locked as much as possible. It comes with: * Secure Boot enabled * A/B boot: the update is written on the other partition, and the system can go back to the previous one if needed * no local access: there is no shell, no SSH, no package manager, nothing can be done with local access to the laptop * disk encryption with the TPM (this allows auto reboot without typing a passphrase, but if someone steals the laptop it will boot and they would be able to attack the system locally as if they were on my LAN) It is not exposed on the Internet, I manage it from my LAN through the Incus API using a certificate for authentication. An unexepcted side effect is that I do not use SSH anymore to manage the infrastructure. No SSH keys to manage, no SSH endpoint to secure, no TERM variable issue. When I need to look inside an instance, I can use `incus exec` to run a command or `incus shell` to open a shell, all through the Incus API. The only SSH in the whole setup is the git over SSH access provided by Forgejo, which is an application feature. # Disposable system containers This is probably the part I like the most. Each instance is created from a fresh Ubuntu cloud image, and cloud-init does everything at first boot: upgrade the system, install podman, write the quadlet files, decrypt the secrets, start the services. I took inspiration from my experience with Qubes OS. When I change anything on an instance (a configuration file, a container, a command, the order of commands), OpenTofu destroys the instance and recreates it from scratch. There is no configuration drift, what runs is exactly what is in git. If someone ever manages to get into a container and leave something behind, it disappears at the next rebuild. Only the data persists, in dedicated Incus volumes: * a volume for podman storage, where the named volumes of the applications live * a volume for the secrets of the instance (its decryption key) * a local cache for the apt packages, to speed up rebuilds As the system part is fully disposable, I could even replace Ubuntu with another distribution without much trouble, the applications run in podman containers anyway. This is heavy: during a change, the component is not available for one or two minutes, the time for it to be rebuilt. But in exchange, every time I make a change, I am sure it will still work at the next boot, and I know I can rebuild the whole system if needed. # Network denied by default All instances are connected to an internal bridge with an Incus ACL that rejects everything by default in both directions, including between two instances. Every allowed network connection is declared in the OpenTofu code, with a description of why it exists. For example, the reverse proxies can reach the exact port of each backend, Prometheus can reach the metrics exporters, and every instance can send its logs to Loki, but Nextcloud cannot talk to Vaultwarden. The goal is to protect the systems from each other. If one service gets compromised, the attacker should not be able to jump easily to the others. There are two VMs for the most sensitive parts: * the haproxy VM, which is the only machine connected to the physical network and receives the Internet traffic (DMZ) * the CI runner, because CI jobs run arbitrary code, it also features Docker in Docker to run nested containers which required some privileges The isolation of a container is a bit weaker than a VM, but it is acceptable for my threat model. For the parts most exposed to untrusted input, I preferred a hypervisor level security. ## Outgoing traffic Most instances have no direct access to the Internet. A dedicated instance acts as the single exit point and provides: * an HTTP proxy with a list of allowed domains, everything else is refused (the domain list is shared by all consumers, this is acceptable here) * a caching proxy for apt packages (apt-cacher-ng) * local mirrors (pull-through caches) for the container registries I use: Docker Hub, GitHub and Codeberg All the instances use these for their packages and container images, and the few applications that need to reach the Internet go through the filtering proxy. The few exceptions that cannot go through a proxy (for example ACME certificates for the public reverse proxy, the CI runner, the Uptime Kuma probes or the backups to S3) are explicit and documented ACL rules. The benefit is that even if a service gets hacked, it will be complicated to exfiltrate data or download a second stage payload from an arbitrary server. Rebuilding twenty instances does not put load on the upstream mirrors or waste network bandwidth. ## Incoming public traffic The public traffic goes through several layers. ``` Internet | v haproxy VM (CrowdSec bouncer with nftables) | raw TCP + PROXY protocol v caddy (public) --> public services ``` The haproxy VM only forwards raw TCP, it never sees cleartext traffic and does not hold any certificate or private key. It uses the PROXY protocol, a small header added at the beginning of the connection that carries the original client IP, so caddy knows who is really connecting, even with NAT and containers in the middle. On the same VM, the CrowdSec firewall bouncer drops connections from IPs with a bad reputation, directly in nftables, before haproxy even accepts the connection. CrowdSec reads the logs of caddy, Forgejo and Nextcloud from Loki to detect bad behavior, and also uses community blocklists. This only works because the real client IP is known all along the way. Then, caddy terminates TLS and forwards the requests to the services. I discovered caddy with this project, and I find it is a great piece of software. The documentation is very good, it always had the features I needed, and the configuration is easy to understand. ## Private access through the VPN There are two separate caddy instances: one for public websites, and one for the sites only reachable over my VPN. This way, even if the public caddy gets hacked, the VPN only services are not affected. The VPN caddy only listens on the Tailscale address of the host, and also checks that the client IP is in the VPN range. The admin tools (Alertmanager, Uptime Kuma, Kanboard...) have no public endpoint at all. Some tools are exposed on the Internet, but their admin interface or admin API is only reachable through the VPN. Even if there is a vulnerability in the admin API, this will not be exploitable. For example, Nextcloud is available both publicly and over the VPN, but its admin API is only reachable through the VPN. Immich, which I use for photos and videos (I wrote about it this year) goes further. Publicly, it only answers to share links, so I can share content with people outside. Everything else like browsing the library or uploading photos and videos, goes through the VPN. This is a security measure, the full application is not exposed over the Internet. For the VPN sites, I wanted real TLS certificates that are not "self signed". I run my own certificate authority (CA), which delivers certificates through ACME, the same protocol used by Let's Encrypt. Its root certificate is installed on all the instances and on the devices (phones, computers, tablets), so every component knowing this root CA recognizes the certificates. No more `curl -k`! # Secrets management No secret are committed in clear in the repository. Secrets are encrypted using sops and age, and stored in git next to the service files. => https://github.com/getsops/sops sops GitHub page Each instance has its own age private key, generated once and stored in the volume `secrets`, it never leaves the server. The secrets of a service are encrypted for that instance key and for my workstation keys only. This means only the person or machine that needs to know a secret can read it: a compromised Nextcloud cannot read the Vaultwarden secrets, even with a full copy of the repository. At boot, the secrets are decrypted on the disposable root disk, so they are regenerated at every rebuild. A git pre-commit hook running gitleaks also blocks a commit if a secret is found in the history, just in case. # Automatic updates The whole stack updates itself: * Incus OS updates itself as an immutable image * Ubuntu instances use unattended-upgrades, and a full upgrade is done when the instance is created, to be sure it starts with the latest packages (I still need to figure how to upgrade the whole base image instead of upgrading it in-place) * the podman containers are updated by `podman auto-update` (as a service), which pulls new images for the configured tag and rolls back if the new container fails to start I chose tags following a major version (for example `postgres:18-alpine`), so major upgrades are a deliberate one line change in git. So far, I never had any issue after a container updated itself automatically. I subscribed to releases RSS feeds for each project I am using, to know quickly if they are breaking changes I need to deal with or a new major version. # Backups There are two layers of backups: * ZFS snapshots of the various volumes, done automatically by Incus, for a quick recovery (daily snapshots kept two weeks for the data) * restic backups of the data volumes every day to an S3 storage, encrypted on the client side If an attacker takes control of the restic instance, they can get the access to the bucket and delete the backups. To protect against this, the bucket has versioning and protections in place, so even if the data is deleted, I can restore it during a certain number of days. The bucket is not in immutable mode though, because restic modifies files at each run and this would break it. I explained this setup in an older blog post: => https://dataswamp.org/~solene/2024-10-19-secure-backups-with-s3.html Secure backups with S3 # Monitoring and logs The whole stack is monitored: * Prometheus collects metrics from every instance, from the reverse proxies, haproxy, CrowdSec and Incus itself * a vector sidecar in every instance sends the systemd journal to Loki, so all the services logs are in one place * Alertmanager sends me a notification when a service fails or is not running * Grafana allows me to explore data (including some cool dashboards to view caddy, haproxy, crowdsec or incus activity) Loki is also the data source of CrowdSec, so logs serve both for debugging and for security. # Security In security, we often talk about three criteria: confidentiality (only the right people can read the data), integrity (the data and systems are not modified without permission) and availability (the service works when you need it). Here is how the different layers help with them in my setup. You only see those criterias when working on certified environments, but I like how they frame the various aspects of cyber security. ## Confidentiality * secrets are encrypted in git, each instance can only read its own secrets * the egress filtering makes data exfiltration hard * the network ACL prevents a compromised service from reaching the others * the admin interfaces are only reachable over the VPN * the edge VM holds no TLS key, TLS is only terminated inside the network * backups are encrypted by restic before leaving the server ## Integrity * Incus OS uses Secure Boot and an immutable system * packages and images only come from known sources through the local caches * no manual change is done, git is the only source of truth * the instances are rebuilt from git, an intruder cannot persist on the root disk (to some extent) ## Availability * CrowdSec blocks known bad IPs, weird requests and brute force attempts * automatic updates reduce the time a vulnerability stays open, this is relevant nowadays * ZFS snapshots allow a fast rollback, restic allows a full restore * the S3 versioning protects the backups against deletion * monitoring and alerts tell me quickly when something goes wrong * the whole infrastructure can be rebuilt from the repository The main limit for availability is that everything runs on a single machine at home, if it dies or if my Internet connection goes down, everything is down. And a rebuild of the public caddy or haproxy cuts the public traffic for a minute or two. This is fine for me. Incus OS have cluster features, but this goes too far for my setup. # Challenges It was not easy, here are the biggest challenges I faced. * sops in cloud-init: sops is not packaged in distributions I wanted to use, I ended up running sops itself as a podman container to decrypt the secrets at boot * the Incus ACL: the mechanism is not always simple to understand, with everything denied in both directions, connections between two instances must be allowed on both the egress and the ingress side * the ordering done by OpenTofu: as Internet access depends on the proxy and the registry mirrors, they must be up before anything else, and for example the CrowdSec bouncer needs the CrowdSec local API to be running to start * CrowdSec in general was very complicated to get right * getting the source IP of a request with NAT and the service running in a container, this was magically solved by haproxy and the PROXY protocol, once I learned about that PROXY protocol * the layers of reverse proxies and the HTTP headers up to the final application, it was particularly hard with PeerTube and Nextcloud to get them to use the correct domain * some applications do not like to work with tw https://dataswamp.org/~solene/2026-10-15-selfhosting-infrastructure-incus-os.html