Quack,
Toushirou is unfortunately loosing one of its network link and we need
to reorganize the whole network configuration. It's a little bit touchy,
especially with the remote storage decryption procedure so we may need
to do a trip at the datacenter to finish the setup. We should have
warned you a little bit earlier (even if this bad news came not long
ago).
We're working on it this very day, so expect some disturbances and
potentially a bit of downtime (most web services).
Regards.
\_o<
--
Marc Dequènes (Duck)
Quack,
Happy New year! (slightly late, sorry)
We hope you're still safe and doing well.
=== Memory Problems on Orfeo ===
Orfeo is old and lacks memory but only recently did we hit problems.
During the last few weeks we had to restart failed services and we
started investigating. Nothing major changed on our side but the
antivirus's update method had changed to be more efficient at the cost
of more memory usage (clamav's ConcurrentDatabaseReload) and we though
that was it. Unfortunately something seems to be leaking memory, we
could not find the source and decided to reboot. Orfeo took a very long
time to come back (03:10-08:40 CET) and we though it lost. Everything's
working fine again but we'll be monitoring the situation.
=== Misc News from end of 2021 ===
- LDAP improvements: the deployment was totally reworked to be simpler,
with better security rules and now all three servers are replicating
each other (n-way sync) which means there is no single point of failure
anymore.
- all machines were upgraded to Debian Bullseye successfully.
- Matrix experimentation is going on well: the web interface improved
and communities were replaced by spaces; ask for an invitation on IRC if
you wish to join the DuckCorp space.
- the webmail was upgraded bringing dark mode (to preserve your eyes)
and other improvements/fixes.
- we use certbot to generate Let's Encrypt certificates and it's missing
an [important feature](https://github.com/certbot/certbot/pull/7244). so
far we used the patch in this ticket but it does not seem it will be
included. Until this is resolved one way or another we're now using a
simpler workaround patch to ease the maintenance and we're also
evaluating using another tool (acme.sh for e.g.).
Hugs.
\\_o<
--
Marc Dequènes (Duck)
* song by Kyoson Asahara and Shinpei Nakayama
(https://en.wikipedia.org/wiki/Teru_teru_b%C5%8Dzu)
Quack,
We hope you're safe and doing well.
=== Improved Mailing-Lists ===
We upgraded our mailing-lists to Mailman 3. It's not just about the
shiny UI, the underlying mail routing daemon is better in many way.
We plan to add LDAP authentication but integration requires extra work
since it's not available out of the box.
=== New System for Users' DNS Primary Zones (aka DNS4Tenants) ===
Banya, our GPG Mail Command gateway, is soon going to retire. This was
inspired by Debian tools and made to be very secure, but unfortunately
sending a properly GPG-signed/encrypted mail with most MUAs is still not
that trivial, making zone updates more painful that it should be. The
script doing the mail handling and DNS update was also far too brittle
and maintenance over time proved problematic.
We're replacing the current system with something easier to use without
compromising security: tenants can now edit their zones in a git
repository of their choice and under 5 minutes a script should pick the
changes, check the zone validity, send errors to the user, and publish
the result if all is fine. It might not sounds like it but the new
script is by far simpler and smaller. The git repository will be fetched
using HTTPS and can be hosted anywhere (including DC). If you wish to
keep your zone hidden then it needs to be accessible using the script's
SSH key; most forges allow that. At DC this is also possible but we're
working on a better solution.
DC and MP zones are now managed using the new system and available in
our openinfra repo. We'll contact users to handle the migration.
=== Web Key Directory Service ===
If you have an email in @dc.o or @mp.o you can now make your GPG key
available using this protocol if you use them in one of your UIDs. It is
an alternate way of fetching keys: the owner of the domain certifies it
is a valid email address and the key association. It is supported by
more and more MUAs, and after all the security problems discovered in
Key Servers' implementations, it should both improve security and
usability.
This comes with an automated way to setup and update the association, so
you start using it right away.
We can also provide this service for hosted domains.
And some documentation:
https://users.duckcorp.org/index.php/Services/WKD
=== Misc news ===
* Matrix:
* the server is working well; we still have made no decision about
IRC mapping.
* Documentation is now available:
https://users.duckcorp.org/index.php/Services/Matrix
* IRC: thanks to Mikachu's suggestion we now have a DNSBL configured and
it seems to be working well against the recent SPAM; it is also used for
antispam (weighted)
* DNSSEC: work has been done both upstream and on our side to fix
various problems. Full automation is not yet complete but making
progress.
* Backup: Pilou added an extra disk for the backup on Nicecity. We have
a basic backup but the target system is still WIP
Hugs.
\\_o<
--
Marc Dequènes (Duck)
Quack,
We wish you a better year (a bit late, sorry).
Many hugs to you.
=== Toushirou's Move ===
We need to move Toushirou since Nerim is closing the room where it is
currently housed. Pilou is kindly handling the physical part of it since
I cannot do that remotly from Japan.
We'll send more detailed info about the disturbance and timing.
=== IRC Upgrade ===
We recently upgraded our IRC servers. This version of the software
reached end of upstream support and that will make switching to the soon
to come new Debian release easier anyway. The documentation was updated
accordingly:
https://users.duckcorp.org/index.php/Services/IRC
Highlights:
- problems with older IRC clients
(https://docs.inspircd.org/faq/#why-does-my-client-not-show-mode-changesoppe…);
on Thorfinn IRSSI was fine but weechat was upgraded to a backport (I
tried using the /upgrade command but it crashed so avoiding a restart is
probably not possible)
- SASL/EXTERNAL method is now available, practical with certfp
(recommended) to get security and autologin (TLS required); read the doc
on the wiki for more info
- STS is supported to suggest IRC clients to switch to TLS
=== IRC Policy Changes
To increase security, privacy and ease fighting disturbing people that
do not forget to visit us every year we are changing some requirements.
MilkyPond will become a TLS-only network starting early March. Nowadays
IRC clients on all OSes/devices all support TLS and it's just one single
checkbox to check so it should not be difficult for non technical people
to switch. The available port list will not change, only insecure
connections won't work anymore.
Registering an account is not compulsory but recommended too; channel
owners might require it though. Using certfp and SASL/EXTERNAL it is
very easy to automagically login to your account while connecting. It's
the same recommendation you can find on freenode and other major
networks and it's not very difficult to setup (see on the wiki).
#DuckCorp is our admin channels. We ought to always be able to discuss
peacefully even when the IRC network is being spammed, so we are going
to require an account to join this channel at the same date we make TLS
compulsory. This does not affect other channels.
=== The world of Matrix ===
We wanted to experiment running a Matrix server to provide an
alternative method of communication. We still use IRC but not everyone
find it practical, XMPP is available too but lost traction, several of
us use Signal but even if the clients are opensource the server is not
and it's totally centralized, so we're now providing a new method for
those who wish to try. It works fine so far but some features are still
missing.
The wiki is not yet updated but if you wish to try you can connect with
your LDAP uid/password on the "milkypond.org" domain (no need to
register with us, your Matrix account is created at first login) with:
- the MilkyPond Web UI: https://matrixonweb.milkypond.org/
- Quaternion on a GNU/Linux desktop
- Element on Android (available on F-Droid)
(other clients exist but they were not tested)
We also experimented with a bridge to join communication between the
#MilkyPond and #DuckCorp IRC channels and their Matrix counterpart,
sorry for the noise, but that did not work well because of several
limitations and the fact end-to-end encryption was enabled on Matrix
thus preventing the discussions to flow towards IRC which does not
support such encryption. The goal was to prevent splitting the community
but maybe that's not such a good idea. Ideas welcome.
If you lack a service that would make this time less painful, please
tell us.
Be safe.
\_o<
--
Marc Dequènes (Duck)
Quack,
Hope you are well.
Despite this difficult time we've done some work but the announcement
was delayed.
=== Reboot ===
Well, it's high time we finally do it. There was some concerns with GRUB
recently, so it was delayed again but it's really needed. I'll try to do
it next week and will be announcing status on IRC. The whole infra is
concerned so I'll roll things up one by one and will inform of the
outages along with the announcement. If all goes well that will be
brief.
=== Webmail and GPG ===
If you're using the webmail to send encrypted mails, we've been
upgrading the software and signing (and signature verification) is now
working fine too.
=== IRC ===
We had a bunch of spammers dumping nasty things earlier this year. We
loaded new plugins in the IRC software to ease filtering them out but it
caused some disruption, sorry for that.
IRC on Web was upgraded and the UI had several problems fixed and some
missing functions added.
=== Security ===
Public-facing services now all use Let's Encrypt, there is no need to
import DC's custom CA anymore. We are working to add DANE support to
more services (currently only SMTP) as a secondary method to validate
you're going to the right machine.
=== Misc ===
In Stuffcloud Talk the TURN/STUN server was not working well, is ti now
fixed.
The MineTest server was unused for some time and was removed.
Be safe.
\_o<
--
Marc Dequènes (Duck)
Quack,
=== Improved Webmail ===
As you must have seen the webmail was upgraded and much more modern
theme which can now be used on a mobile phone. The old theme is still
available and it can be changed in your settings if your preferred it.
There are many little improvements and fixes which are not immediately
visible but should improve our daily life. The support for Mailvelope,
to be able to encrypt/sign mails from your browser, without disclosing
your key to the server, is supposed to be improved but I was not able to
enjoy it yet. I will come back to you when I have more time to test but
feedback is welcome too.
To secure your login TOTP is also available. After the upgrade there is
a small problem though: the QR code does not show up in the new theme,
please switch back temporarily to the Larry theme in your settings. More
details on how to use it in the documentation:
https://users.duckcorp.org/index.php/Services/Mail#Using_a_Web_Interface
=== Web Security ===
We are adding extra security features to our hosted web hosts and
especially for our services we are limiting the content and features a
website can use (using various headers and especially CSP).
On of the consequence is the webmail will not accept unsecure links. I
just discovered it should be possible to upgrade the link to a secure
version on the fly, which should solve most usability problems; I will
work on this soon. If it is blocked, then it is unsafe and your browser
logs should say so.
If you want to protect your personal website, please contact us; safe
settings are applied everywhere but only fully-managed services will be
upgraded automagically, as knowledge of the site's content is needed to
protect without breaking.
=== Stuffcloud Talk ===
Now that Toushirou has a new body, and also some software upgrades, this
video chat application is working fine so far, enjoy.
=== DNS Security ===
Our main zones are already secured (DNSSEC) but we changed the
underlying software (OpenDNSSEC to Bind inline-signing). This simplifies
a lot of things even if it is not yet perfect.
With these changes we are able to secure dynamic zones easily and the
DDNS zone is now safe. It also makes things possible on the PKI side
(see below),
=== PKI ===
Some time ago we began use Let's Encrypt to generate HTTPS certificate
that would be trusted by all major browsers. This is working fine and
the automation is great. Nevertheless we still kept our non-web services
under the umbrella of our custom DuckCorp CA. There are initiatives to
secure the web but they all rely on the infamous self-appointed CAs.
Despite this, bringing services and user software to use secure
connections is important, so we decided to use Let's Encrypt for more
services. Currently this only affects SMTP servers but more will follow.
This should improve the trust the big providers/corporation on the
Internet give to our server (like our mails landing on your recipient's
SPAM box for no good reason). The work done on the DNS allows us to
deploy Let's Encrypt certificates for non-web services (using the DNS
challenge).
Using an external CA is not necessarily giving away the extra security
we had with our custom CA. There is a method (DANE) to publish our
certificates for each service via the DNS on a secure zone, which we do
have. We were already publishing them for non-web services but it had to
be reimplemented to work with Let's Encrypt. We should then have the
best of both worlds.
=== Mail Security ===
We've reinforced the security level of various services (TLS settings…)
and especially the SMTP and IMAP part.
For SMTP we are now using a set of servers protected by DNSSEC with
published certificates (DANE). We also advertise a policy to enforce
secure connections (MTA-SAS, the HSTS of the mail).
=== AntiSPAM ===
Previously we were quite unforgiving with badly configured servers
sending badly formatted or unresolvable introduction (HELO), but a lot
of providers have very bad practices and this caused some difficulties
to use certain services. We are now using Rspamd filters which are
weighting these problems among other things to discover if a mail is
really a SPAM and we have relaxed the previous SMTP rules.
=== IPv6 broker ===
We lost our dedicated IP block quite some time ago and there was an
expensive way to get another. As funny as this service was, not so many
people cared about IPv6 and now that most Internet connections have it
included it has been a very long time anyone asked for it. This was de
facto over but now it's official.
These are quite important changes to our infra. If you encounter any
problem, please let us know.
And enjoy the autumn leaf viewing too! :-)
\\_o<
--
Marc Dequènes (Duck)
Quack,
=== A Word on NiceCity ===
So the machine is back online and works fine. Supervision is working too
though it still need some tweaking.
The work to rebuild the backup is in the todolist now.
=== System Upgrades ===
We've been preparing upgrades to Debian Buster and the work has already
started. Nicecity was rebuilt on Buster directly. Elwing recently
migrated too, so we were able to adapt our deployment rules.
Not everything is ready for all machines to migrate yet but we're going
to start migrating machines starting from now. Because of kernel changes
and also some network settings that needs to be adapted, this means at
least one reboot, maybe more.
We'll try to minimize downtime but expect some short disruptions.
You can follow the steps on IRC.
\\_o<
--
Marc Dequènes (Duck)
Quack,
Let's begin with the bad news.
=== Credentials and potential LDAP info leak ===
Our MediaWiki instances configuration, as all the rules of our infra are
opensource. Unfortunately certain secrets were not protected as they
should have been:
* database credentials: MariaDB is not available from the outside, so
it's not a big deal
* contact emails: we try to hide all emails to avoid SPAM; webmaster,
HappyPeng and myself are affected
* wiki secret key: used to generate entropy when a proper source is not
available, which is not our case
* LDAP service account: this is a nasty part as most user information
might be accessed; we do not keep more than needed to run the service
but realnames and email addresses could have been listed; all
credentials are safe and no modification is possible though
We did not find any suspicious activities in the logs, but that's
difficult to assess.
So obviously before sending this email we remedied the situation. We
also plan to limit how this account can be used even more.
Deeply sorry for this mistake :-/.
=== Progress on the new hardware ===
Nicecity's hardware gave some difficulties and a new disk was needed. It
is now almost fully deployed with Ansible, so we should soon be able to
use it in production.
Toushirou-NG's deployment is done and we're regularly syncing it. We're
keeping it up-to-date with current production and are preparing the
migration steps.
=== StuffCloud login Security ===
Two factor authentication (2FA) using TOTP and U2F has been activated
and tested for some time now. We did not have any problem with both
method, so we strongly suggest you try it.
The documentation has been updated to give more information on how to
set this up: https://users.duckcorp.org/index.php/Services/StuffCloud
=== Wiki Migration ===
The shared wiki (wiki.duckcorp.org), running on MoinMoin, is going to
migrate soon to MediaWiki in order to avoid having to maintain two
systems.
=== Mail and Sync User ===
Some users prefer to sync emails on their machine, which can be handy if
you travel, wish to access them offline and later have all your
modifications propagated. It can also be useful as a backup mechanism.
On problem with most tools is their inability to "move" mails (instead
of "copy"), which prevent then to use the 'Junk' folder
A simple solution is now available, described here:
https://users.duckcorp.org/index.php/Services/Mail#Retraining_for_feature-l…
=== Web and PHP ===
We were already investigating using PHP FPM instead of the embedded
Apache mod_php. This is more secure and flexible as you can run the
wanted number of workers using a dedicated UNIX user, thus avoiding one
instance to read files from the others.
It happens that we activated HTTP2 in the past and some related fix
unfortunately required to change the Apache MPM, which is not possible
with mod_php, or to abandon HTTP2, so this changed priorities a bit.
The full migration has just been done for all PHP-enabled vhosts, so
tell us if you encounter any problem.
Hope you enjoyed the Hanami!
\\_o<
--
Marc Dequènes (Duck)
Quack,
Many hugs for this new year!
Pilou and I are coming to FOSDEM (and some side-events), so if you're
around do not be shy and say quack.
=== Plans for the Future ===
No crazy plans for this new year yet. We're focused on replacing the
aging hardware and finishing automating our deployments to simplify our
job. There's a few ideas when we get extra power on the new hardware,
but that's for another post.
Nicecity is small and could not keep up with the load of the Monitoring
system, so Pilou kindly is upgraded her. We should transplant her
consciousness soon and restart both monitoring and backup on the new
hardware.
There is no direct user impact but be sure to backup your critical data
on your own just in case.
The work on Toushirou-NG is ongoing, this is coupled with the effort to
Ansibilize our infra which made quite a lot of progress. So we're
basically able to reinstall the host fully with Ansible modulo one or
two small glitches. Data are being synced with the current host too.
Before proceeding to the replacement we need to check that all services
work fine, this is WIP.
DNSSEC (signed DNS zones) is a very nice security feature but
unfortunately managing the keys and their replacement over time is quite
a hassle to say the least. We're currently using OpenDNSSEC and it works
but the setup is a tad complicated. the DNS server (Bind9) has made
quite some progress on this front. We're now using the Debian backports
to take advantages of some new features and we're evaluating switching
to using Bind9 tooling directly.
=== Mail ===
We did some work around the mail filtering system (SIEVE):
- activated spamtest: if you do not use the provided include script to
move SPAM into the Junk box becaue you need more customization, then you
should be interested in this extension. Instead of parsing the headers
yourself, which could break if we change the system, tune the
sensitivity… this extension provides an interface with a normalized
score directly
- activated vacation-seconds: allows more granularity for the vacation
settings
- on the webmail we replaced the unmaintained Roundcube 'sieverules'
plugin by 'managesieve'; this is prepare for the future Debian version
with an improved Roundcube and SIEVE plugin. Currently they are equally
incomplete and buggy, so it should not change anything for you.
- we recently switched from the deprecated dovecot-antispam Dovecot
extension to IMAPSIEVE, which does a similar job but cleaner and more
flexible by far. It currently implements the same exact behavior, so no
user change. In the future Debian version the old extension would not
work, so better be prepared.
The documentation was updated accordingly:
https://users.duckcorp.org/index.php/Services/Mail
We also activated the IMAP metadata extension which is used by some mail
clients to store server and folder custom information. It "might" be
useful, but it's cheap anyway so why not enable it.
=== Cleanup ===
We're continuing to reevaluate our services to focus on the important
things and to be able to gather resources for new projects.
The webstats, fetchmail and feed2imap services were unused, so they were
removed.
\\_o<
--
Marc Dequènes
Quack,
=== Mail System Redundancy ===
Our old antispam system has been shutdown and Toushirou has been
promoted to full MX1/MDA/MAA. This means we now have two machine
receiving mails, filtering spam and synchronizing with each-other (SIEVE
rules and antispam learning are synced too).
I just updated the DNS records so that
{smtp,mail,imap,pop,sieve}.duckcorp.org now balance between the two. So
if one machine is down, you should be able to retry to reach your mail
on the second machine.
The webmail is currently not redundant but we're working on it.
The mailing-lists will stay on Orfeo as there is currently no way to
make this service redundant.
=== Note about the Antispam ===
We had some user questions so I wanted to clarify a few things.
The new learning system is very similar to the previous one. The big
change is: from now on all learning is shared between users. This means
you benefits from SPAM caught by other users but it makes the learning
less personalized. It is difficult to evaluate what's best but per-user
learning made the system very complex and was very resource intensive so
we decided to try the shared method. Many well known big companies also
use shared training, so that new users don't start with a blank system
which would need weeks or months to be well trained and it seems to work
quite well.
Another important point is how to train the system. We simplified the
old system and the new one does not have a web interface for users
anyway. The 'Junk' folder (your mail client may translate the name) is
used to collect discovered SPAM. If you move mails into it you inform
the system it is a spam it should have caught. If you move a mail out of
it then you inform the system it made a mistake and this is not a spam.
To get detected SPAM automatically put into the 'Junk' folder there is a
default SIEVE filter. But if you added your own filters, then you need
to include the default rule into your configuration as explained here:
https://users.duckcorp.org/index.php/Services/Mail
This was done to allow users to override the default rule if they wanted
another workflow.
We're wondering if this has any real application and thinking about
making this rule compulsory (processed before any user filters). Your
input on this matter is welcome.
\\_o<
--
Marc Dequènes (Duck)