Categories
Uncategorized

Proxmox server Lexar NM790 NVME SSD crash

As an IT nerd I try to use tech to its max and having a Proxmox install in use to maximize use of resources on a laptop server. The laptop is use mainly its for low energy consumption and the build in UPS (aka the laptop battery πŸ™‚ ).

That server hosts a shared storage to keep my photos and make automated backups of those photos. So 2 years ago I upgraded the server to a 2Tb NVME disk of Western Digital and had no issues at all with that.

This year I started to review my photo store on the PC which I use for photo editing and came to the conclusion that the servers disk space was to small to store all the photos I have. So it was time to look for a bigger disk. With the Black Friday deals on their way I also checked some reviews of 4Tb disks on a budget and with decent performance. Which resulted in me buying a Lexar NM790 4Tb for a nice price (thanks for the Black friday deals πŸ™‚ ).

So I migrated the Proxmox OS to it and tested it if it would boot on a different laptop. In the end the migration was fairly painless. The copy speed of the data (1.4Tb!) was around 350Mb/s which was quite impressive. There was a catch though (otherwise I would not write this blog πŸ™‚ ).

The server would crash after a day or so (or sometimes faster) with mixed ext4 errors like:

  • kmmpd: 185 comm kmmpd-dm-16: Error writing to MMP block
  • EXT4-fs error in ext4_reserve_inode_write : Journal has aborted

Not really good for a server which is constanly online for internet proxy and network security purposes. So I needed to find answers how to fix it.. And in the end I found some info with the same trouble with Ubuntu on https://askubuntu.com/questions/1521014/lexar-nm790-4tb-ssd-keeps-crashing-my-fresh-install-of-ubuntu-24-04

There should be a newer firmware for the disk fixing the problem, but adding a windows 10 install to the test laptop and trying the correct tools of Lexar did not result in a firmware update. So I will revert to the other solution provided to stop the disk going into sleep mode which is apparently not working properly with Linux :-(.

I will update the blog post once the server is up and running again with the new disk and it survives a few days of uptime.

The solution will be to add:

nvme_core.default_ps_max_latency_us=0Β to the boot line for the kernel.

We should edit the file /etc/default/grub and update the line containing GRUB_CMDLINE_LINUX_DEFAULT and make it look like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvme_core.default_ps_max_latency_us=0"

Then after updating /etc/default/grub, run the following command:

sudo update-grub
Categories
Blog

Shaping up KumoMTA

POstfix Logo

I have been working in the email marketing industry for over 15 years and I have installed and configured most well known mass-mailers in market. Starting off my professional journey as an Open Source solutions engineer I started implementing Postfix and Exim solutions which are fine for low to mid volume email send outs, but you get in trouble when sending over 100k of mails per hour, let alone trying to send millions of mails in a day.

I will admit that back in the days I have worked on multi postfix installs on one server to create separate queues in order to separate mail streams, but even with that IO lead to be the main bottleneck of this solution.

The next step up in the MTA business back then (2010-ish) were PowerMTA from Port25 and Momentum from Message Systems. Today both solutions are owned by Bird creating a nice vendor lock-in for mass mailing solutions.

As an upgrade to the Postfix solution we migrated to PowerMTA which is very capable of sending millions of mails. It can do rate limiting per domain (and even per queue) and it is very easy configurable. It also can respond to bounces in a sensible way and go in back-off mode if too much of the same bounces happen.

Several years later I also worked with Momentum which had a very advanced way to respond to bounces of ESPs. It was called adaptive delivery and a very nice feature which was miles ahead of PowerMTA to respond clever based on the actual bounce response.

For example if Gmail gives a bounce notification when sending a mail: “Our system has detected an unusual rate”. That means that you need to slow down your sending. Momentum was able to then slow down the sending speed or even suspend send out for 30 minutes or more.

And this is the part I love the most since we come at the technical part of email deliverability, which is making email delivery an easier job. You don’t not need to look at logs anymore in order to change the sending rate or move messages in a slower sending queue. With the adaptive delivery properly setup you can concentrate on the actual content of client emails for even better delivery.

Fast forward to 2022, when I started to implement a small scale email platform based on Postfix and I intended to use a custom made Postfix container configuration with dkim and rate-limit support. Even though that it could do most of the things (in a technical sense) that I required for improved delivery compared to a stock Postfix, those deliverability rules still required manual configuration.

With that in mind I went looking for other solutions and at a certain moment I found KumoMTA . After experimenting with it for some time to find out what KumoMTA was capable off, I joined the KumoMTA Discord in december 2023. KumoMTA replaced the Postfixes used in the small scale email platform to at least remove the manual work to update rate limits.

KumoMTA is a very promising mass-mailing solution for all ESPs, but it still is a relatively young project not having all the bells and whistles which PowerMTA and Momentum have. For me and my project it ticked all features I need and especially the shaping automation is something I was very happy about.

In discussions on Discord with Mike Hillyer about the shaping rules and how to make them, we both were on the same page that in 2024 the rules to react to bounce messages is actually common ground. Most big ESPs like Gmail, Outlook, Yahoo and for example Orange.fr do have postmaster pages explaining what each bounce notification stands for.

Mike’s way of explaining that the MTA is a commodity solution:
“The MTA is like a hotel’s plumbing: it better work, but no-one is picking their hotel based on the plumbing.”

With that concept in mind I started to build on the shaping automation helper in KumoMTA and worked on a shaping.toml file which tries to follow up on the most occurring bounce messages with appropriate measures. To make it the community shaping file in the KumoMTA package.

I started off with the big ones like Gmail, Yahoo and Outlook.com and then I added a few others which I came across while delivering mail. I am very proud to now say that in the latest dev builds you will be able to enable the community shaping.toml file to deliver emails better with KumoMTA.

If you are not running the latest versions of KumoMTA then you can still use the file by simply downloading the file from https://github.com/KumoCorp/kumomta/blob/main/assets/community/shaping.toml and use it in your own shaping.toml configuration. It should work in most versions of KumoMTA. I use it myself on the 2024.06.10.140609.74a0f2be -dev version.

Questions can be asked on the KumoMTA Discord and then specifically checkout ‘Solmea’.

Categories
Blog

KumoMTA

Recently I have played around with the Open Source mass mailer KumoMTA. And off course when using such a tool you want to see statistics on how the mail servers is performing.

For this I installed a logstash instance on the KumoMTA server and configured it to accept http requests. With that I was able to configure KumoMTA to inject logs using webhooks into logstash.

And with that logstash was the middleman to inject the data into Elasticsearch. Then lastly the Kibana UI was required to make some nice dashboards to show statistics on the mail performance.