DC-Users

dc-users@lists.duckcorp.org

1 participants
82 discussions

Summer upgrades
by DuckCorp Admin Team 17 Aug '19

17 Aug '19

Quack, === A Word on NiceCity === So the machine is back online and works fine. Supervision is working too though it still need some tweaking. The work to rebuild the backup is in the todolist now. === System Upgrades === We've been preparing upgrades to Debian Buster and the work has already started. Nicecity was rebuilt on Buster directly. Elwing recently migrated too, so we were able to adapt our deployment rules. Not everything is ready for all machines to migrate yet but we're going to start migrating machines starting from now. Because of kernel changes and also some network settings that needs to be adapted, this means at least one reboot, maybe more. We'll try to minimize downtime but expect some short disruptions. You can follow the steps on IRC. \\_o< -- Marc Dequènes (Duck)

1 0

Spring renewal
by DuckCorp Admin Team 25 Apr '19

25 Apr '19

Quack, Let's begin with the bad news. === Credentials and potential LDAP info leak === Our MediaWiki instances configuration, as all the rules of our infra are opensource. Unfortunately certain secrets were not protected as they should have been: * database credentials: MariaDB is not available from the outside, so it's not a big deal * contact emails: we try to hide all emails to avoid SPAM; webmaster, HappyPeng and myself are affected * wiki secret key: used to generate entropy when a proper source is not available, which is not our case * LDAP service account: this is a nasty part as most user information might be accessed; we do not keep more than needed to run the service but realnames and email addresses could have been listed; all credentials are safe and no modification is possible though We did not find any suspicious activities in the logs, but that's difficult to assess. So obviously before sending this email we remedied the situation. We also plan to limit how this account can be used even more. Deeply sorry for this mistake :-/. === Progress on the new hardware === Nicecity's hardware gave some difficulties and a new disk was needed. It is now almost fully deployed with Ansible, so we should soon be able to use it in production. Toushirou-NG's deployment is done and we're regularly syncing it. We're keeping it up-to-date with current production and are preparing the migration steps. === StuffCloud login Security === Two factor authentication (2FA) using TOTP and U2F has been activated and tested for some time now. We did not have any problem with both method, so we strongly suggest you try it. The documentation has been updated to give more information on how to set this up: https://users.duckcorp.org/index.php/Services/StuffCloud === Wiki Migration === The shared wiki (wiki.duckcorp.org), running on MoinMoin, is going to migrate soon to MediaWiki in order to avoid having to maintain two systems. === Mail and Sync User === Some users prefer to sync emails on their machine, which can be handy if you travel, wish to access them offline and later have all your modifications propagated. It can also be useful as a backup mechanism. On problem with most tools is their inability to "move" mails (instead of "copy"), which prevent then to use the 'Junk' folder A simple solution is now available, described here: https://users.duckcorp.org/index.php/Services/Mail#Retraining_for_feature-l… === Web and PHP === We were already investigating using PHP FPM instead of the embedded Apache mod_php. This is more secure and flexible as you can run the wanted number of workers using a dedicated UNIX user, thus avoiding one instance to read files from the others. It happens that we activated HTTP2 in the past and some related fix unfortunately required to change the Apache MPM, which is not possible with mod_php, or to abandon HTTP2, so this changed priorities a bit. The full migration has just been done for all PHP-enabled vhosts, so tell us if you encounter any problem. Hope you enjoyed the Hanami! \\_o< -- Marc Dequènes (Duck)

1 1

Happy New Year!
by Marc Dequènes (duck) 11 Jan '19

11 Jan '19

Quack, Many hugs for this new year! Pilou and I are coming to FOSDEM (and some side-events), so if you're around do not be shy and say quack. === Plans for the Future === No crazy plans for this new year yet. We're focused on replacing the aging hardware and finishing automating our deployments to simplify our job. There's a few ideas when we get extra power on the new hardware, but that's for another post. Nicecity is small and could not keep up with the load of the Monitoring system, so Pilou kindly is upgraded her. We should transplant her consciousness soon and restart both monitoring and backup on the new hardware. There is no direct user impact but be sure to backup your critical data on your own just in case. The work on Toushirou-NG is ongoing, this is coupled with the effort to Ansibilize our infra which made quite a lot of progress. So we're basically able to reinstall the host fully with Ansible modulo one or two small glitches. Data are being synced with the current host too. Before proceeding to the replacement we need to check that all services work fine, this is WIP. DNSSEC (signed DNS zones) is a very nice security feature but unfortunately managing the keys and their replacement over time is quite a hassle to say the least. We're currently using OpenDNSSEC and it works but the setup is a tad complicated. the DNS server (Bind9) has made quite some progress on this front. We're now using the Debian backports to take advantages of some new features and we're evaluating switching to using Bind9 tooling directly. === Mail === We did some work around the mail filtering system (SIEVE): - activated spamtest: if you do not use the provided include script to move SPAM into the Junk box becaue you need more customization, then you should be interested in this extension. Instead of parsing the headers yourself, which could break if we change the system, tune the sensitivity… this extension provides an interface with a normalized score directly - activated vacation-seconds: allows more granularity for the vacation settings - on the webmail we replaced the unmaintained Roundcube 'sieverules' plugin by 'managesieve'; this is prepare for the future Debian version with an improved Roundcube and SIEVE plugin. Currently they are equally incomplete and buggy, so it should not change anything for you. - we recently switched from the deprecated dovecot-antispam Dovecot extension to IMAPSIEVE, which does a similar job but cleaner and more flexible by far. It currently implements the same exact behavior, so no user change. In the future Debian version the old extension would not work, so better be prepared. The documentation was updated accordingly: https://users.duckcorp.org/index.php/Services/Mail We also activated the IMAP metadata extension which is used by some mail clients to store server and folder custom information. It "might" be useful, but it's cheap anyway so why not enable it. === Cleanup === We're continuing to reevaluate our services to focus on the important things and to be able to gather resources for new projects. The webstats, fetchmail and feed2imap services were unused, so they were removed. \\_o< -- Marc Dequènes

1 0

Mail Revamping #2
by DuckCorp Admin Team 27 Jun '18

27 Jun '18

Quack, === Mail System Redundancy === Our old antispam system has been shutdown and Toushirou has been promoted to full MX1/MDA/MAA. This means we now have two machine receiving mails, filtering spam and synchronizing with each-other (SIEVE rules and antispam learning are synced too). I just updated the DNS records so that {smtp,mail,imap,pop,sieve}.duckcorp.org now balance between the two. So if one machine is down, you should be able to retry to reach your mail on the second machine. The webmail is currently not redundant but we're working on it. The mailing-lists will stay on Orfeo as there is currently no way to make this service redundant. === Note about the Antispam === We had some user questions so I wanted to clarify a few things. The new learning system is very similar to the previous one. The big change is: from now on all learning is shared between users. This means you benefits from SPAM caught by other users but it makes the learning less personalized. It is difficult to evaluate what's best but per-user learning made the system very complex and was very resource intensive so we decided to try the shared method. Many well known big companies also use shared training, so that new users don't start with a blank system which would need weeks or months to be well trained and it seems to work quite well. Another important point is how to train the system. We simplified the old system and the new one does not have a web interface for users anyway. The 'Junk' folder (your mail client may translate the name) is used to collect discovered SPAM. If you move mails into it you inform the system it is a spam it should have caught. If you move a mail out of it then you inform the system it made a mistake and this is not a spam. To get detected SPAM automatically put into the 'Junk' folder there is a default SIEVE filter. But if you added your own filters, then you need to include the default rule into your configuration as explained here: https://users.duckcorp.org/index.php/Services/Mail This was done to allow users to override the default rule if they wanted another workflow. We're wondering if this has any real application and thinking about making this rule compulsory (processed before any user filters). Your input on this matter is welcome. \\_o< -- Marc Dequènes (Duck)

1 0

Mail Revamping
by Marc Dequènes (duck) 07 Jun '18

07 Jun '18

Quack, === New Antispam === Today is a special edition about mail. From September 2007 we've been using DSPAM, a very nice piece of antispam technology. Instead of neverending rules fiddling this system properly learns to recognize unwanted mails. Unfortunately it has been more than four years that maintenance stopped and we've been having a hard time finding a replacement. We believe we did and now is time to give a chance to the new system to prove it. Thanks a lot to all the DSPAM contributors, that was really awesome. Our new system is Rspamd. The new system uses a mix of solutions to score mails, including headers analysis, Bayesian rules, adaptative greylisting (activates if mail appears somewhat spammy), RBLs… We've selected rules that would not leak mail content onto third parties. We've also decided to drop per-user learning, as it really makes the architecture more complex and consume more resources (early rejection would no more be possible), but current tests shows it should work out fine. Currently Rspamd and DSPAM are working concurrently (for filtering and learning) but we intend to stop DSPAM very soon. Mail filtering should continue to work as usual. I updated the documentation accordingly: https://users.duckcorp.org/index.php/Services/Mail Outgoing mails are now checked by Rdpamd too. Please let us know if you have any problem. The system also needs more input to learn but it should not take long. Don't forget to move in/out the `Junk` folder SPAM/HAM mails to teach the system. === Some words about the future === We're currently transforming Toushirou into a full mail server (MX1), so when Orfeo is in maintenance or a problem occurs, we should be able to read our mails and receive new ones. We needed more storage for Toushirou-NG and Pilou offered new disks, thanks a lot :-). We're working on bringing Toushirou's new body to life. Have a Lot of FUN! \_o< -- Marc Dequènes

1 0

We shall get there some day
by DuckCorp Admin Team 19 May '18

19 May '18

Quack, === Incidents === Orfeo was down from 2018-04-20 to 2018-04-24 due to a crash. We don't know if this was a kernel bug or a hardware failure. It is working again well fortunately. Hivane had a problem with a switch on 2018-05-09. It is working again well after a reboot. They plan to change the switch after this sign of weakness. === Future of the Hardware === We're working on a body replacement for Toushirou. We acquired a machine, not brand new but still a better one, it should improve the performance of some demanding webapps like StuffCloud. Deployment is taking more time than expected but we're on it. We're also reloading the idea of possible home-hosting. This experiment might give a future to Orfeo which is not in a safe location. Also the recent crash is a sign the machine is really getting too old and need replacement. At the moment the expenses are small, but you're welcome to contribute. We did not pay yet for the new machine for Toushierou; the rest of the expenses can be seen here: https://users.duckcorp.org/index.php/DcExpenses === IRC SPAM === To mitigate SPAM we worked on the channel settings but this was not sufficient. We now require users to authenticate to services and as the attacks respawn from time to time (even if you don't see anything now) this is going to stay. We also recommend you connect via SSL for security and privacy, using port 6680. === SSH Keys Update Bug === A bug in the synchronization script removed all SSH keys from user accounts. Keys registered in the LDAP were reinstalled but manually added keys, which usually are preserved, were lost. Due to an unfortunate problem accessing the backup server, we could not re-add them. If you need keys added, just contact us using a secure channel. === Open Infrastructure === Most services in our infrastructure are now managed via Ansible and the rules are publicly accessible: https://projects.duckcorp.org/projects/dc-admin/wiki We're working hard on automating and opening the remaining bits. The Admin wiki content was moved (and often improved) in the Redmine wiki or the Users's wiki except for very few sensitive data. This wiki was closed. Also most admin discussions/decisions now openly take place either on Redmine or IRC on #DuckCorp. Feel free to contribute! === Misc news === * the IM Gateway was not used anymore, so we stopped the service * SVN/TLA/CVS repositories were not used anymore, support in shell and Redmine was removed * the Redmine timesheet plugin was removed; it was broken by several bugs (like https://github.com/arkhitech/redmine_timesheet_plugin/issues/49) and not well maintained anymore * Yoshi has a new skin! Also it is not anymore asking people to register (but you might do so if you wish) Hugs. \\_o< -- Marc Dequènes (Duck)

1 0

Happy New Year!
by DuckCorp Admin Team 05 Jan '18

05 Jan '18

Quack, Hope you had a nice celebration time :-). === Plans for the Future === One of the main priority this year is to replace Orfeo and Toushirou's bodies, as they are getting quite old (and slow). If you have any advice or opportunity, please tell us. There's also work underway to make service more reliable and we'd like to revive the monitoring which is currently down because Korutopi is no more. We'll continue evaluating our service to close down obsolete ones and focus energy on more interesting things. Feel free to suggest improvements. === Open Infrastructure === There's ongoing steps to make our work more visible and transparent. DuckCorp is almost an Open Infrastructure now, with only few documentation bits still hidden. Many things are still manually managed on the servers, so involuntarily hidden, but with the switch to Ansible for configuration management, this is improving quickly. You can have a look at the project wiki here: https://projects.duckcorp.org/projects/dc-admin/wiki And the code / main README: https://projects.duckcorp.org/projects/dc-admin/repository All the bits are Free Software of course. We welcome contributions. === Security Fixes === In light of the recent processor-related failures, we'll be rebooting the machines with new kernels soon, when all fixes are available. The exact time will be announced on IRC. === IRC Upgrade === We already planned to change the IRC softwares which became unmaintained. TLS linking support was broken but happily all traffic is confined in the Hivane network so no big deal. Also the services regularly "forget" about channel settings, even for so called permanent channels, which is really annoying. We'd also like to promote use of TLS in order to improve the network's privacy. We will do this step by step. Recently many SPAM attacks happened on public networks and being a small isolated network did not help. Enforcing TLS and/or identifying to NICKSERV could be a very efficient way to block all SPAM out. The services will soon be shutdown, we can live without it. One of the two servers will be put out of rotation too to be rebuilt with the new software. Please use `irc.milkypond.org` to connect or you may end-up on the empty and unstable server being built. The switch to the new network will be advertised on the old network and the remaining server shutdown to allow people to reconnect automagically to the new one. \_o< -- Marc Dequènes (Duck)

1 0

Maintenance: Toushirou moving to PA3
by Marc Dequènes (duck) 19 Sep '17

19 Sep '17

Quack, 2017-09-21 early afternoon (CEST) Toushirou will be shutdown and moved in a new home. During this time various services will be unavailable, mainly: all websites except the webmail, secondary DNS and mail servers, DDNS, FTP. Godspeed Toushirou! \_o< -- Marc Dequènes

1 0

Subject: What happened while you were on the beach?
by DuckCorp Admin Team 04 Sep '17

04 Sep '17

Quack, === Backup Back Online === It took a while, but thanks to Pilou's hard work we have full backup coverage again. If you'd like some data to be excluded from backup, either because it's useless or because you prefer not, you just need to add an empty '.nobackup' file in the directory to ignore (recursively). It's the same behavior as before, so if you already setup things, nothing changed. === More Changes after Stretch Migration === Noone asked about the Webdesk, so we're going to remove it very soon. We have a new cute IRC on Web as previous one was unmaintained, and broken after migration, enjoy! Former stats using Awstats were utterly broken and noone noticed; Piwik has been working well since a while and was promoted instead. Please use webstats.dc.o (no more webstats-ng.dc.o). Ask if you need your website to be integrated into the new system. The photos website (using Gallery2) was utterly broken after migration; we have no replacement yet but we're looking into it, sorry. Blogs were never used and broken after migration; they were removed. === A few Security Changes === Users with shell access should review their SSH keys (both stored and authorized); DSS and low grade RSA will soon be removed (advice: use ed25519). Persons (not only users) managing a DNS zone should use DNS CAA RR to protect from domain hijacking (please read https://www.isc.org/blogs/certificate-authority-authorization-records/). This is far from a perfect solution but may still help. We can do that for you if the zone is hosted by DC, just ask us. We've activated HTTP2 support on all websites, and this comes with improved ciphers and web settings. Also we plan to ensure all websites, user ones included, are all redirected to HTTPS, no exceptions. === Mail Security when using the Webmail === There is now an option to PGP encrypt/decrypt mails using our webmail (Roundcube) using Mailvelope. It is a browser extension allowing to use your PGP key locally, so it stays stored on your computer and the encryption/decryption occurs on your computer too. Unfortunately it does not handle signatures but let's hope the support in Roundcube improves. You can have a look here: https://www.mailvelope.com/ === Quick NEWS === * 2017-07-29 around 09:00 CEST and for ~2h a DNSSEC problem broke all DNS resolution on duckcorp.org domain due to a bug in OpenDNSSEC; happily noone seem to have noticed \_o< -- Marc Dequènes (Duck)

1 0

What happened while you were on the beach?
by Marc Dequènes (duck) 04 Sep '17

04 Sep '17

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

DC-Users