Coin,
=== News ===
Tōshirō was prepared and successfully installed in its definitive
location today, thanks to Yok, Nefou, and people from Hivane and Sivit.
Moving critical services has started. Our NS and NTP have moved a few
minutes ago. While this is transparent for NTP, you may have to do some
changes in you zone settings (see the following chapter if you have such
service).
Mail and Web migrations are being prepared. Mails won't move until a few
days, because a few architecture changes and improvements will take
place, and because much testing is needed. You'll be warned when this is
gonna happen. Web pages should be more easily configured on the new box,
but because of the big amount of data, a solution needs to be found to
avoid taking years with the ADSL upload limitations. So, please be
patient.
=== NS Update ===
Tōshirō NS is available on two IPs:
- ns1.duckcorp.org (replacing Orfeo)
- ns2.duckcorp.org
It then ensure a "piece of redundancy", for network failures only.
You can ask for another NS (provided by Hivane) if you need full
redundancy.
When Orfeo is back in a datacenter, another NS would then be available.
A) If you have a master zone hosted:
A.1) for the master zone:
/!\ It is no more possible to edit your zone via shell access. A new
method would be available in the future, but this will have to wait
after the situation is all back to normal. if you need any change, then
ask us via mail or IRC.
If your registrar is Gandi and you gave us technical rights on your
zone, then everything was already made for you. Skip to B.2.
If not, then you should add ns2.duckcorp.org to your NS list for better
redundancy (in the registrar database only, the zone was already updated
by DC admins).
A.2) for the external slave zone(s):
Please update the masters which are now :
- 193.200.42.177
- 80.248.213.245
B) If you have a slave zone hosted:
B.1) for the slave zone:
There is nothing to do.
B.2) for the external master zone:
Please update the IP allowed for transfers with :
- 193.200.42.177
- 80.248.213.245
Moreover, you should add ns2.duckcorp.org to your NS list for better
redundancy (in the zone and in the registrar database).
Beware ! For those who had the unwise idea to use this kind of
configuration for their master zone:
---
@ NS ns1.mydomain.tld.
@ NS ns2.mydomain.tld.
ns1.mydomain.tld. A 1.2.3.4
ns2.mydomain.tld. A 195.5.254.194
---
First, ns2.mydomain.tld. does not exists, then you'll have to change the
server IP each time ns1.duckcorp.org moves. I agree this is a better
name for your domain, but people reading this kind of technical
information would soon understand you've got no real NS behind this
name. Moreover, either you've added the corresponding glue record, and
you are poluting important NS servers, either you didn't and you would
surely experience strange behaviors or even your whole domain be
unavailable. In either case, this is *WRONG*, correct it ! The right
configuration is:
---
@ NS ns1.mydomain.tld.
@ NS ns1.duckcorp.org.
@ NS ns2.duckcorp.org.
ns1.mydomain.tld. A 1.2.3.4
---
=== ... ===
If anything in this mail is unclear or if you need assitance for using
our services while maintenance is in progress, ask us via mail or IRC.
Stay tuned...
--
Marc Dequènes (Duck)
_______________________________________________
DC-Admins mailing list
DC-Admins(a)lists.duckcorp.org
https://lists.duckcorp.org/mailman/listinfo/dc-admins
Coin,
We are on the move.
== The Plan ==
A new machine (Tōshirō), recently acquired, which was on the way to be
installed for production, is going to be moved this very sunday into a
datacenter. This was not the original plan, but we decided to haste
things a bit, and happily found a hosting for it.
Critical services are going to be "displaced" as fast as possible. A few
others will be handled one by one during the following days, because a
lot more configuration needs to be checked, and also lot's of data
moved.
== The First Step ==
As soon as general configuration and critical services are done, we are
going to push the web online again. Then, for those still having access
to their web content: please DO NOT modify or add anything starting from
sunday, or it will probably be lost while we are copying data to the new
server.
== General Recommandations ==
Services may have unvolontary downtimes, please be patient. We are doing
our best, but unexpected problems sometimes happen.
Do not rely anymore on orfeo.duckcorp.org hostname, DO USE the proper
service alias (smtp.duckcorp.org, imap.duckcorp.org, ...) or you will
not be able to access services if they are moved to another machine. It
should already be ok, as we advertised aliases quite a lot ; be warned
we won't advertise any service relocation in the future if the admin
team deem it necessary. If you don't know the proper alias to use, just
ask us, via mail or IRC.
Drink beer ! It helps forget problems, forgive bad people, and is a good
social catalyst ;-).
Stay tuned...
--
Marc Dequènes (Duck)
Coin,
= Where we are =
This afternoon (2007-08-08T15:29:07), without any warning or delay,
Nerim's boss decided to shutdown our main server, Orfeo. Thanks to
Nerim's friends and Hivane[1], the box was recovered in the evening and our
IPs routed to our HQ ADSL. Due to bandwidth limitations, it is no more
possible to provide full services.
Services unavailable :
- Web (including chrooted SSH, Yoshi IRC Stats, and Wotomae)
- FTP
- Radio
= Where we go =
We are investigating possible relocation solutions. We currently believe
it is a matter of days before a hosting solution is found.
Moreover, a new machine was acquired recently. It is not yet ready for
production, and the hosting infrastructure is still being worked on by
our network sponsor Hivane[1], but it should be possible to improve
services availability. More on this topic soon.
= And what now ? =
Now is time to go to bed, we are tired...
[1] http://www.hivane.net/
--
Marc Dequènes (Duck)
Coin,
1) Orfeo's crashes:
A crash happened at 2006-06-29, due to an XFS Kernel bug [1]. This bug
is currently unsolved and may cause further problems. The machine was
restarted at 2006-07-14 before we loose control, a newer kernel was
installed (with lot's of XFS fixes, but not the expected one) and /home
repaired. Since now, things are working fine again, and we are following
the issue closely.
Due to another XFS bug [2] in the installed kernel, we'll have to update
it again soon ; the exact date will be advertised on IRC.
You should not have lost any data, but just in case you're into
problems, we've got backups ; then do not hesitate to ask us if
necessary.
2) PHP5 migration:
As many applications are now switching to PHP5 only, it has been decided
to switch to this new major version, as it is not possible to have both
version 4 and 5 working at the same time.
Around 2006-09-01, the switch will occur. Then you have a full month to
check compatibility of your applications and modify your custom code
accordingly. Packaged softwares will be handled by the Admin Team, so
you should not have to care.
3) IPv6:
Since yesterday, IPv6 connectivity is back. XS-26, our former provider,
was slow to handle the switch to the new IP range (due to the complete
removal of the experimental IP range 3ffe), and service quality was
poor, so we were looking at other opportunities. Happily, Nerim made
some upgrades recently, and native connectivity is now possible.
Please inform us if you find any problem related to this change. Most
services are IPv6 aware, and we are working at having them all work with
Ipv6 properly.
4) Mail Filtering:
Current issues are :
- since a short while, our filtering system has been configured to use
"defanging", meaning SPAMs not killed results in a mail report with
the original Subject and From, and the problematic mail attached as a
RFC 822 attachment. This gives a clearer result with more details,
but it seems it is causing problems for mailboxes splitting, mostly
because custom headers are not repeated in the report mail
- filtering limits in the report differs from the custom field in the
problematic mail attached, weird !
- filtering rules needs to be updated, because less SPAM is filtered
since a few months, suggestions are welcome
We are investigating and working on these issues.
Thanks for your patience, and thank you for using <del>fre</del>
DuckCorp ! ;-)
[1] http://bugzilla.kernel.org/show_bug.cgi?id=5856
[2] http://oss.sgi.com/projects/xfs/faq.html#dir2
--
Marc Dequènes (Duck)
Coin,
Here are some news from the Pond.
1) Mailing-lists outage:
A misconfiguration of our SMTP server was cause of mail subcription
rejects. The applied fix was wrong and every mail between 2006-03-07
early in the morning and 2006-03-08 in the night were rejected (my bad
!). This was worked out as soon as we realized what happened, and both
problems are now fully solved. Apologies.
2) Backup:
The new hard drive was installed in the backup server (whose hardware
was upgraded in the same way). Old backups are safe, and new backups are
now created. Total DC backups are now 67GB ; it was really time for
upgrade ! Moreover, the old hard drive was installed on Elwing for
private storage, but a check found it full of bad blocks !
3) Security concerns about shell access:
To avoid giving shell access to people having no real need for it, some
adjustments were necessary to let you have the same level of service.
a) Personnal web space:
The personnal web spaces are now available through your FTP account into
the 'www-perso-dc' directory. Ask for FTP access if you need.
HurdFr users can access they personnal web spaces into the
'www-perso-hurdfr' directory the same exact way.
The DC photo album upload space is in the same way accessible into the
'photos-perso-dc' directory.
Please note this is only a convenient way to access ressources and a
security improvement ; people having shell access may still manage their
files with their account.
For those who care about security, FTP is not less secure for 2
reasons:
- first, catching the local password gives a relatively short range of
priviledges, incomparable with having a shell account, whatever
priviledges it may be granted. Look at SecurityFocus news to imagine
how many local root exploits exists.
- moreover, you can increase the security level by reading the next
change below (Secure FTP).
b) SQL administration:
For those who fear about losing their shell access and be unable to
manage their database, don't worry, a phpMyAdmin interface has been made
public here:
https://sql.duckcorp.org/
c) Shell access removal:
The following users should contact us to look at their web space ACLs
and dicuss about the needs that would not be fulfiled by the provided
tools before their shell access is removed:
- cedricburnay
- js
- marius
- scop
- valfor
Note that a shell access may be reopened if necessary in the future ;
this is not a punishment, but a security measure.
3) Secure FTP:
You can now connect to our FTP server with a TLS-enabled client, this
would secure your data transfers, and moreover, avoid any password
disclosure, as the TLS mode is enabled before authentication.
Don't forget to get the DC certificate here:
https://www.duckcorp.org/dc/ca.crt
and check the signature here:
https://www.duckcorp.org/dc/ca.crt.asc
(if you trust my key, of course)
Free* tools supporting FTP+TLS:
- on GNU/Linux and GNU/Hurd, the Netkit 'ftp-ssl' package
- on Windows, FileZilla (http://filezilla.sourceforge.net/)
- on Mac OS X, Cyberduck (http://cyberduck.ch/)
* Free like in "free speech"
see http://www.gnu.org/philosophy/free-sw.html
(in french: http://www.gnu.org/philosophy/free-sw.fr.html)
4) Upgrades:
a) mySQL:
First, the database was cleaned, unuseful things were trashed, and a
naming convention was established. If your name is Arnau, GuiHome,
HurdFr or PikaPaf, then your DB users/db-names may have changed to match
the convention ; don't worry, everything was renamed for you and your
website is working as usual.
Furthermore, the database was successfully upgraded from 4.1 to
5.0. Changelog says the most important new features are:
- stored procedures
- triggers
- views
- information schema
- archive storage engine (for historical and audit data)
If this software could stop crashing once and a while, it would be the
best new feature ever dreamed of...
By the way, don't forget to do some database management periodically to
improve access to your data:
- check if no indexes are missing
- use ANALYSE to improve search capabilities
- use OPTIMIZE to clean up unnecessary data
phpMyAdmin can help you do this (with no more than 3 clicks ;-).
b) BitlBee:
BitlBee is an IM gateway through a nice IRC interface, you can only
access it localy (if you have a shell account) at the moment, using port
6668 like you would do for any IRC server.
The new 1.0.1 version was installed, fixing some nasty bugs (full
changes here: http://bitlbee.org/main.php/changelog.html). The irssi
scripts were upgraded too, allowing typing notification in the
statusbar.
You need to /reconnect to the server to benefit from these fixes.
5) Spam Learning system reminder (or not):
To improve detection of spam and avoid false positives, 2 special
mailboxes have been setup. You can send spams which where not detected
to dc-ham(a)duckcorp.org and false positives (if not destroyed) to
dc-ham(a)duckcorp.org to train the system and improve recognition.
/!\ PLEASE TAKE CARE to use the "resend" function of your mailer and
never use "forward", which would result in _yourself_ being considered
as spam or ham /!\
This said, Have a Lot of Fun in the Pond !
--
Marc Dequènes (Duck)
Coin,
First, wanted to wish you lot's of happiness for this new year :-)
News follows:
1) Orfeo illness:
On 2006-01-07 morning, Orfeo suffered from a major hard disk access
problem, leading to most services being unusuable during most of the
day. After some attempts to take back control, a move to the datacenter
was scheduled. In the middle of the afternoon, Orfeo was up and ok
again. Investigations revealed a bug in the XFS subsystem ; a bug is
opened here:
http://bugzilla.kernel.org/show_bug.cgi?id=5856
2) Short SPAM infestation:
Due to a mistake i made on 2006-01-10 when upgrading amavid-new, the
mailer system was working half a day without spam check. Sorry for the
inconvenience.
3) Really available BW:
Orfeo's bandwidth was meant to be 100Mbps in the new datacenter, but a
configuration mistake in the network equipment prevented us from taking
advantage of this nice upgrade. This was solved on 2006-01-16 morning.
4) Backup:
Due to increase of important data to backup, it was obvious the current
storage (60GB) was no more sufficient. Thanks to an anonymous donor, we
have a brand new disk and can now store as much as 300GB. The old disk
will move to Elwing, where the 8.4Go data hard disk was dead, providing
space for services located on Elwing and private data storage. this is
currently work in progress.
May the CoinForce be with you this year !
--
Marc Dequènes (Duck)
Coin,
The year is (almost) over. We are happy services have worked quite well
and hope you're happy with them. Feel free to report problems or leave
comments.
We wish you a Merry X-mas, nice meals and stuff :-)
Here are the last news of the year:
1) Orfeo move:
As you may have noticed, the move was a success. DC DNS zone was updated
quickly and many services were back online quickly. No reboot was
necessary and remaining services were working again during the night.
All DNS zones should have been corrected now. Please remember to use
CNAMEs for web vhosts instead of A records, to avoid delays for future
changes.
By the way, kernel was updated and PaX patch too, solving a w3m problem
called "absurd stack bottom value". No other problems were reported, but
similar behaviors with other programs should be solved too.
2) Web news:
Certificates are now up to date now. Because we were not able to get an
IP block, we are now using a wildcard certificate for each server, thus
fixing browser complaints for all DC vhosts (but not for hosted
projects' vhosts).
Horde configuration was fixed (PHP configuration indeed), solving file
uploads (so you can now import your keys to sign your mails in IMP for
example).
A PHP cache engine, with a good reputation (used to handle the scaring
skyblogs), eAccelerator, was installed. This should improve latency a
wee bit. Feel free to report / comment on it so we can decide if it is
worth running it.
A photo album dedicated to DC users and friends was setup here:
http://photos.duckcorp.org/main.php
Feel free to ask for an account.
3) Mail processing changes:
We are now using the experimental amavisd-new package, as the unstable
one was heavily outdated, fixing several problems, and allowing better
stats and quarantine. The system now complains on bad headers without
rejecting the mail completly, and is now storing intermediate scored
mails depending on their badness (virus, bad header, spam, banned)
during several days (final number would depend on available disk
space). Do not hesitate contacting us if you suspect a good mail was
trashed, quickly, so we can analyse the quarantine and changes settings
if necessary.
4) Chat tools upgrade:
Bitlbee 1.0 was out recently and fixed some issues (a security one, and
several others ; even if no one in here reported problems). Most visible
change is the channel renaming from #bitlbee to &bitlbee to avoid
confusion with OFTC channel name.
IRSSI 0.8.10 was out recently too and installed. If you are using it on
Orfeo, please /UPGRADE now (to use the new binary without
quitting). Many many leaks were fixed, and a recode engine is available,
so pleaaaaaaaase don't use the so bad existing scripts anymore
(/help recode). Beware some script may not behave correctly ; please
report problems affecting system-installed scripts to us.
5) Memory problems:
Recent memory problems are now solved. Services cleanup, tweaks, and
fixes (mostly IRSSI leaks fixes) gave us back an acceptable amount of
memory. This may improve services responsiveness (and thus jam PHP cache
tests too).
6) FTP bandwidth limitations:
As we are now on a bigger internet connection (100Mbps), rate limiting was
removed, and should stay removed if it is not slowering other services.
7) DC standing against EUCD:
As part of DC policy to react on important events, and after a vote of
the board, DC officially stand against DADVSI (as you may have seen in
the main news site). Useful links on the subject was given on the site,
please read them for your own sake.
--
Marc Dequènes (Duck)
Coin,
1) Orfeo moving:
Orfeo is moving from Ivry to LDCOM datacenter on thursday end of
afternoon. An almost dedicated VLAN is reserved, which is good, but host
IP is changing and some DNS problems may occur during a short while. We
did not manage to get an IP block, thus secure web hosts would still
trigger warnings in browsers.
Beware Orfeo may be restarted 2-3 times for necessary tests while
changing kernel and network configuration.
2) Minor other changes:
A) IRC server and services:
These were updated recently and major bugs, mostly due to buggy IPv6
support, were reported, kindly fixed by upstream authors, and new
packages were created, tested, and installed (corresponding Debian
packages are available in the DuckCorp repository: hyperion and theia).
Remind all members are welcome to join irc.milkypond.org. Most useful
channels being #MilkyPond (community channel) and #DuckCorp
(organization channel), and that you may create one yourself.
B) Certificates:
DuckCorp CA public certificate is available through:
https://www.duckcorp.org/dc/ca.crt
and signature:
https://www.duckcorp.org/dc/ca.crt.asc
You may have seen web certificates have expired during this weekend.
They are soon being regenerated and you will be able to check their
validity using the CA certificate if you have already signed my key
(if not, a keysigning party can be arranged).
If you did not understand what i said before, or why, feel free to ask
questions.
C) PHP:
The fileinfo module was added with some PEAR classes, but turck-mmcache
was removed (broken on Debian testing).
Have a Lot of Fun !
--
Marc Dequènes (Duck)
Coin,
1) New mailing list:
As the DC website is quite static and you don't use it regulary, one
suggested me to create a mailing list to keep a better link with our
users. This mailing list is private, mostly intended for announcement
(new services, planned maintenance, outages, ...), but you may also
use it to request help.
You were automatically subcribed to this list, and may at any moment
unsubcribe. But please do not forget you will then be unaware of changes
affecting services you use, and then should not complain about
unavailable critical services or data loss.
2) Maintenance:
a) MySQL upgrade:
MySQL was upgraded from 4.0 branch to 4.1 last monday. All applications
have proved to continue working perfectly. Notice the password format
has changed, some users were then asked for help, and every password was
changed unchanged (meaning only the internal format changed by
reentering the same exact password into the database), as this may cause
login problems with certain database drivers (rails one for exemple).
b) Secondary mail servers:
Hivane was providing MX2 for several zones with a server which is now
offline due to loss of housing. Then, mx0.hivane.net was replaced in all
the affected zones with mx2.hivane.net. As a security, mx3.hivane.net
(which is the same host with a different AS path) was added as MX3 with
the same priority.
3) New services:
a) mail server auth and security:
smtp.duckcorp.org is now providing securised connections through
SMTP+TLS (using the standard port 25) or SMTPS (port 465). Using such a
connection, the server allow user authentication, and would then relay
mails from anywhere on the universe (useful for roaming users).
b) Webdesk:
https://webdesk.duckcorp.org/ is hosting Horde, a framework including
several web applications, among them is the famous IMP mail client.
Don't be afraid, the traditionnal webmail application (SquirrelMail)
won't be removed. This is a quite powerful application, but still having
some minor bugs (sometimes IMP load fails: log out and in to solve,
filebrowser ask for login again: horde automatic login does not work and
was deactivated). Have fun using / discovering it.
A user once asked me for this feature, here it is.
4) Housing move:
Nerim Ivry datacenter, where Orfeo is housed, is being sold, and every
computer is moving to the LDCOM datacenter (Courbevoie). In a few days /
weeks, Orfeo will then be shut down for a few hours. As soon as a more
accurate schedule is available, you will be notified.
Have a Lot of Fun !
--
Marc Dequènes (Duck)
