Quack,
=== Mail System Redundancy ===
Our old antispam system has been shutdown and Toushirou has been
promoted to full MX1/MDA/MAA. This means we now have two machine
receiving mails, filtering spam and synchronizing with each-other (SIEVE
rules and antispam learning are synced too).
I just updated the DNS records so that
{smtp,mail,imap,pop,sieve}.duckcorp.org now balance between the two. So
if one machine is down, you should be able to retry to reach your mail
on the second machine.
The webmail is currently not redundant but we're working on it.
The mailing-lists will stay on Orfeo as there is currently no way to
make this service redundant.
=== Note about the Antispam ===
We had some user questions so I wanted to clarify a few things.
The new learning system is very similar to the previous one. The big
change is: from now on all learning is shared between users. This means
you benefits from SPAM caught by other users but it makes the learning
less personalized. It is difficult to evaluate what's best but per-user
learning made the system very complex and was very resource intensive so
we decided to try the shared method. Many well known big companies also
use shared training, so that new users don't start with a blank system
which would need weeks or months to be well trained and it seems to work
quite well.
Another important point is how to train the system. We simplified the
old system and the new one does not have a web interface for users
anyway. The 'Junk' folder (your mail client may translate the name) is
used to collect discovered SPAM. If you move mails into it you inform
the system it is a spam it should have caught. If you move a mail out of
it then you inform the system it made a mistake and this is not a spam.
To get detected SPAM automatically put into the 'Junk' folder there is a
default SIEVE filter. But if you added your own filters, then you need
to include the default rule into your configuration as explained here:
https://users.duckcorp.org/index.php/Services/Mail
This was done to allow users to override the default rule if they wanted
another workflow.
We're wondering if this has any real application and thinking about
making this rule compulsory (processed before any user filters). Your
input on this matter is welcome.
\\_o<
--
Marc Dequènes (Duck)
Quack,
=== New Antispam ===
Today is a special edition about mail. From September 2007 we've been
using DSPAM, a very nice piece of antispam technology. Instead of
neverending rules fiddling this system properly learns to recognize
unwanted mails. Unfortunately it has been more than four years that
maintenance stopped and we've been having a hard time finding a
replacement. We believe we did and now is time to give a chance to the
new system to prove it. Thanks a lot to all the DSPAM contributors, that
was really awesome.
Our new system is Rspamd. The new system uses a mix of solutions to
score mails, including headers analysis, Bayesian rules, adaptative
greylisting (activates if mail appears somewhat spammy), RBLs… We've
selected rules that would not leak mail content onto third parties.
We've also decided to drop per-user learning, as it really makes the
architecture more complex and consume more resources (early rejection
would no more be possible), but current tests shows it should work out
fine.
Currently Rspamd and DSPAM are working concurrently (for filtering and
learning) but we intend to stop DSPAM very soon. Mail filtering should
continue to work as usual. I updated the documentation accordingly:
https://users.duckcorp.org/index.php/Services/Mail
Outgoing mails are now checked by Rdpamd too.
Please let us know if you have any problem. The system also needs more
input to learn but it should not take long. Don't forget to move in/out
the `Junk` folder SPAM/HAM mails to teach the system.
=== Some words about the future ===
We're currently transforming Toushirou into a full mail server (MX1), so
when Orfeo is in maintenance or a problem occurs, we should be able to
read our mails and receive new ones.
We needed more storage for Toushirou-NG and Pilou offered new disks,
thanks a lot :-). We're working on bringing Toushirou's new body to
life.
Have a Lot of FUN!
\_o<
--
Marc Dequènes
Quack,
=== Incidents ===
Orfeo was down from 2018-04-20 to 2018-04-24 due to a crash. We don't
know if this was a kernel bug or a hardware failure. It is working again
well fortunately.
Hivane had a problem with a switch on 2018-05-09. It is working again
well after a reboot. They plan to change the switch after this sign of
weakness.
=== Future of the Hardware ===
We're working on a body replacement for Toushirou. We acquired a
machine, not brand new but still a better one, it should improve the
performance of some demanding webapps like StuffCloud. Deployment is
taking more time than expected but we're on it.
We're also reloading the idea of possible home-hosting. This experiment
might give a future to Orfeo which is not in a safe location. Also the
recent crash is a sign the machine is really getting too old and need
replacement.
At the moment the expenses are small, but you're welcome to contribute.
We did not pay yet for the new machine for Toushierou; the rest of the
expenses can be seen here:
https://users.duckcorp.org/index.php/DcExpenses
=== IRC SPAM ===
To mitigate SPAM we worked on the channel settings but this was not
sufficient. We now require users to authenticate to services and as the
attacks respawn from time to time (even if you don't see anything now)
this is going to stay.
We also recommend you connect via SSL for security and privacy, using
port 6680.
=== SSH Keys Update Bug ===
A bug in the synchronization script removed all SSH keys from user
accounts. Keys registered in the LDAP were reinstalled but manually
added keys, which usually are preserved, were lost. Due to an
unfortunate problem accessing the backup server, we could not re-add
them. If you need keys added, just contact us using a secure channel.
=== Open Infrastructure ===
Most services in our infrastructure are now managed via Ansible and the
rules are publicly accessible:
https://projects.duckcorp.org/projects/dc-admin/wiki
We're working hard on automating and opening the remaining bits.
The Admin wiki content was moved (and often improved) in the Redmine
wiki or the Users's wiki except for very few sensitive data. This wiki
was closed.
Also most admin discussions/decisions now openly take place either on
Redmine or IRC on #DuckCorp.
Feel free to contribute!
=== Misc news ===
* the IM Gateway was not used anymore, so we stopped the service
* SVN/TLA/CVS repositories were not used anymore, support in shell and
Redmine was removed
* the Redmine timesheet plugin was removed; it was broken by several
bugs (like
https://github.com/arkhitech/redmine_timesheet_plugin/issues/49) and not
well maintained anymore
* Yoshi has a new skin! Also it is not anymore asking people to register
(but you might do so if you wish)
Hugs.
\\_o<
--
Marc Dequènes (Duck)
Quack,
Hope you had a nice celebration time :-).
=== Plans for the Future ===
One of the main priority this year is to replace Orfeo and Toushirou's
bodies, as they are getting quite old (and slow). If you have any advice
or opportunity, please tell us.
There's also work underway to make service more reliable and we'd like
to revive the monitoring which is currently down because Korutopi is no
more.
We'll continue evaluating our service to close down obsolete ones and
focus energy on more interesting things. Feel free to suggest
improvements.
=== Open Infrastructure ===
There's ongoing steps to make our work more visible and transparent.
DuckCorp is almost an Open Infrastructure now, with only few
documentation bits still hidden. Many things are still manually managed
on the servers, so involuntarily hidden, but with the switch to Ansible
for configuration management, this is improving quickly.
You can have a look at the project wiki here:
https://projects.duckcorp.org/projects/dc-admin/wiki
And the code / main README:
https://projects.duckcorp.org/projects/dc-admin/repository
All the bits are Free Software of course. We welcome contributions.
=== Security Fixes ===
In light of the recent processor-related failures, we'll be rebooting
the machines with new kernels soon, when all fixes are available. The
exact time will be announced on IRC.
=== IRC Upgrade ===
We already planned to change the IRC softwares which became
unmaintained. TLS linking support was broken but happily all traffic is
confined in the Hivane network so no big deal. Also the services
regularly "forget" about channel settings, even for so called permanent
channels, which is really annoying.
We'd also like to promote use of TLS in order to improve the network's
privacy. We will do this step by step.
Recently many SPAM attacks happened on public networks and being a small
isolated network did not help. Enforcing TLS and/or identifying to
NICKSERV could be a very efficient way to block all SPAM out.
The services will soon be shutdown, we can live without it. One of the
two servers will be put out of rotation too to be rebuilt with the new
software. Please use `irc.milkypond.org` to connect or you may end-up on
the empty and unstable server being built. The switch to the new network
will be advertised on the old network and the remaining server shutdown
to allow people to reconnect automagically to the new one.
\_o<
--
Marc Dequènes (Duck)
Quack,
2017-09-21 early afternoon (CEST) Toushirou will be shutdown and moved
in a new home. During this time various services will be unavailable,
mainly: all websites except the webmail, secondary DNS and mail servers,
DDNS, FTP.
Godspeed Toushirou!
\_o<
--
Marc Dequènes
Quack,
=== Backup Back Online ===
It took a while, but thanks to Pilou's hard work we have full backup
coverage again.
If you'd like some data to be excluded from backup, either because it's
useless or because you prefer not, you just need to add an empty
'.nobackup' file in the directory to ignore (recursively). It's the same
behavior as before, so if you already setup things, nothing changed.
=== More Changes after Stretch Migration ===
Noone asked about the Webdesk, so we're going to remove it very soon.
We have a new cute IRC on Web as previous one was unmaintained, and
broken after migration, enjoy!
Former stats using Awstats were utterly broken and noone noticed; Piwik
has been working well since a while and was promoted instead. Please use
webstats.dc.o (no more webstats-ng.dc.o). Ask if you need your website
to be integrated into the new system.
The photos website (using Gallery2) was utterly broken after migration;
we have no replacement yet but we're looking into it, sorry.
Blogs were never used and broken after migration; they were removed.
=== A few Security Changes ===
Users with shell access should review their SSH keys (both stored and
authorized); DSS and low grade RSA will soon be removed (advice: use
ed25519).
Persons (not only users) managing a DNS zone should use DNS CAA RR to
protect from domain hijacking (please read
https://www.isc.org/blogs/certificate-authority-authorization-records/).
This is far from a perfect solution but may still help. We can do that
for you if the zone is hosted by DC, just ask us.
We've activated HTTP2 support on all websites, and this comes with
improved ciphers and web settings. Also we plan to ensure all websites,
user ones included, are all redirected to HTTPS, no exceptions.
=== Mail Security when using the Webmail ===
There is now an option to PGP encrypt/decrypt mails using our webmail
(Roundcube) using Mailvelope. It is a browser extension allowing to use
your PGP key locally, so it stays stored on your computer and the
encryption/decryption occurs on your computer too. Unfortunately it does
not handle signatures but let's hope the support in Roundcube improves.
You can have a look here:
https://www.mailvelope.com/
=== Quick NEWS ===
* 2017-07-29 around 09:00 CEST and for ~2h a DNSSEC problem broke all
DNS resolution on duckcorp.org domain due to a bug in OpenDNSSEC;
happily noone seem to have noticed
\_o<
--
Marc Dequènes (Duck)
Quack,
=== Backup Back Online ===
It took a while, but thanks to Pilou's hard work we have full backup
coverage again.
If you'd like some data to be excluded from backup, either because it's
useless or because you prefer not, you just need to add an empty
'.nobackup' file in the directory to ignore (recursively). It's the same
behavior as before, so if you already setup things, nothing changed.
=== More Changes after Stretch Migration ===
Noone asked about the Webdesk, so we're going to remove it very soon.
We have a new cute IRC on Web as previous one was unmaintained, and
broken after migration, enjoy!
Former stats using Awstats were utterly broken and noone noticed; Piwik
has been working well since a while and was promoted instead. Please use
webstats.dc.o (no more webstats-ng.dc.o). Ask if you need your website
to be integrated into the new system.
The photos website (using Gallery2) was utterly broken after migration;
we have no replacement yet but we're looking into it, sorry.
Blogs were never used and broken after migration; they were removed.
=== A few Security Changes ===
Users with shell access should review their SSH keys (both stored and
authorized); DSS and low grade RSA will soon be removed (advice: use
ed25519).
Persons (not only users) managing a DNS zone should use DNS CAA RR to
protect from domain hijacking (please read
https://www.isc.org/blogs/certificate-authority-authorization-records/).
This is far from a perfect solution but may still help. We can do that
for you if the zone is hosted by DC, just ask us.
We've activated HTTP2 support on all websites, and this comes with
improved ciphers and web settings. Also we plan to ensure all websites,
user ones included, are all redirected to HTTPS, no exceptions.
=== Mail Security when using the Webmail ===
There is now an option to PGP encrypt/decrypt mails using our webmail
(Roundcube) using Mailvelope. It is a browser extension allowing to use
your PGP key locally, so it stays stored on your computer and the
encryption/decryption occurs on your computer too. Unfortunately it does
not handle signatures but let's hope the support in Roundcube improves.
You can have a look here:
https://www.mailvelope.com/
=== Quick NEWS ===
* 2017-07-29 around 09:00 CEST and for ~2h a DNSSEC problem broke all
DNS resolution on duckcorp.org domain due to a bug in OpenDNSSEC;
happily noone seem to have noticed
\_o<
--
Marc Dequènes
Quack,
Debian has been released! Omedetou!!!
=== Global Thermonucl^WUpgrade ===
Joshua will shortly upgrade all hosts and schedule reboot to use the new
kernel. We will announce disruptions on IRC.
We have already decided to deprecate a few services (see below) and more
may come. Maintaining unused softwares is painful and if noone cares
then we'd better redirect our efforts elsewhere. If we announce a
deprecation and you're still an active user or the replacement does not
suit your need, then please contact us.
=== Webmail ===
Squirrelmail is unmaintained, has security issues, and will be removed
during the previously announced upgrades. Someone even told me it fell
broken recently, so I guess it's a sign retirement has been postponed
long enough.
Horde works, and should continue to, but does not add much now that
RounCube made quite some progress (and more after upgrade) and the extra
apps are not very useful compared to what StuffCloud offers; it will be
reevaluated, so tell us if you wish it to stay.
=== StuffCloud recent upgrade ===
Well, not so recent now. For a few hours the file view was utterly
broken after an upgrade due to a bug, sorry for that.
A new 'Circles' application allowing private/shared groups has been
added. In the past we add an application for personal groups but it was
not well maintained and this one makes this feature available again and
more (and should be better maintained as it is a core-app).
=== Quick NEWS ===
* [2017-04-30] updated TLS certificates => services restart (very short
disruption)
* VCS:
** It has never been properly advertized but rcs*.duckcorp.org are
deprecated and will be removed in September; please use vcs* addresses
instead
** Git protocol enabled for Bip project, thanks to Trou for pointing
this was missing
* IRC on Web: The qwebirc instance available on
https://irconweb.milkypond.org has been updated to use TLS while
connecting to the IRC server
* NTSX: Noone is using it and now tethering, plans, and local free WIFI
(when you travel for eg) improved quote a bit so it is useless now and
is gonna be removed soon
Don't forget your umbrella.
\_o<
--
Marc Dequènes (Duck)
Quack,
Long time no quack, busy being ill…
=== Mail Rejection ===
As announced we had to change Orfeo's IPv4 recently. The new one was
unused, so no previous spammer owned it, but the big corps have
wittingly decided to consider it having a bad reputation by default.
Also we discovered a user recently had his account compromised and it
was used to relay SPAM.
So we're stripping naked and leaking their asses to get our mails
accepted, but it's taking time and maybe we'll need to give more of
ourselves to get it. So please bear with us and report your problems
with as much details as possible so we can debug and act accordingly
(even if much of it is out of our control).
=== StuffCloud Ultimate Upgrade ===
This is an upgrade we've been postponing for too long. At some point
having no fixes and no security support is really bad (even if it's not
that great anyway). Also the next Debian release is coming and this
software would not work anymore.
So we've decided to switch from OwnCloud to NextCloud, and upgraded step
by step. This was not easy to get something working in the end but here
we are. Please tell us if you experience any difficulties.
The list of apps is quite similar, so no data loss. The interface gained
a bit of slowlessness too, charming. But there's fixes and a few new
features you may like.
Please note there is a NextCloud Android app on F-Droid, so while the
old OwnCloud one still works, you should probably replace it.
Also, to ease communication with the rest of the world, we decided to
use letsencrypt to provide the HTTPS certificate for this service. This
means it is now easier to give URLs to external people for sharing (no
certificate to add to the browser manually). We may use letsencrypt for
other websites in the future, but no decision made yet.
=== Out of Backup ===
Our previous backup system (Bacula) had some shortcomings and we
ended-up having all backup expired and purged while we were busy
elsewhere. We believe the current behavior is flawed and this software
has proved to be slightly complicated to maneuver, so it is time to have
some fresh air.
We're working on deploying a new system, but in the meanwhile we're
naked and if you have critical data I would advise you to do your own
backup for the time being. We're really sorry of the situation and hope
to have it solved really soon.
Grrr Grrr Grrr
\_o<
Happy new year !
=== Planned maintenance ===
In order to apply updates, all servers will be restarted during the
night of Thursday 2 and Friday 3 February.
Restarts will be performed one by one in the following order: Orfeo,
Thorfinn, Jinta, Toushirou.
All services will be affected, hopefully services with some redundancy
will remain available:
- mail (redundancy: mx1.duckcorp.org, mx4.duckcorp.org)
- websites
- DNS (redundancy)
- IRC (redundancy: irc1.duckcorp.org, irc2.duckcorp.org)
- Bip
- IRC2IM gateway
- Jabber
- PostgreSQL
- MySQL
- LDAP (redundancy)
- NTP
sorry for the inconvenience.
=== Orfeo: accommodation ===
Thanks to Hivane, Orfeo has now a more durable location. Again, big
thank to Hivane for the hosting, you may want to consider making a
donation to them [1].
=== Orfeo: new IPv4 address ===
Soon, the main IPv4 address of Orfeo will be changed from 193.17.192.211
to 193.200.43.105. The new IPv4 is already added, we still need to
update the DNS configuration.
[1] http://www.hivane.net/donate.html
--
Pilou