Sunday, 26 December 2010

Basic Debian Server Maintenance

1) Subscribe to the Debian Security mailing list. Check here : You can also subscribe to other relevant mailing lists if you want but make sure to subscribe to the debian-security-announce. I recommend that you use an address that is checked regularly.

2) Once you receive a security announcement saying that a package has been updated, log onto your server and:

Update your package database:
sudo aptitude update

If there is an update for one (or many) of the packages you use the output will finish by:
Reading package lists… Done

Current status: 1 update [+1]

Perform the actual upgrade:
sudo aptitude safe-upgrade
If the upgrade was successful it should finish by:
Current status: 0 updates [-1]

Check to see if currently running processes or daemons are using an older version of the updated packages/libraries:
sudo checkrestart

It will tell you if processes are using old versions of files, what are the scripts and argument you can use to make sure they are restarted with the proper version. (Note that if you are connected by SSH and checkrestart tells you to reload it you will also need to logout/login again for this to be effective). You will notice that on a Debian (and other Linux) servers you almost never need to restart the whole server.

Note: If you don't have checkrestart, you may just need to install it:
sudo aptitude debian-goodies
3) If you're using Tripwire (and you should, if not take a look at this guide), run a check and update the database so it doesn't report those upgrades as errors. Just go back to this guide if you don't remember how.

Thursday, 23 December 2010

Using Fail2Ban for SSH on custom ports

You migth have already setup Fail2ban properly following this guide. But if you're not using standard ports for your services, Fail2ban could identify the threat but not properly block the offending user.

When Fail2ban identify an IP as a possible threat it will modify your firewall (iptables) to block that IP from accessing your service. So if IP has tried too many times to access your server through SSH but failled, fail2ban will "tell" iptables to block this IP from accessing your server through your "ssh" port. Which translate to port 22. But what if your SSH server is on port 44000?

Following the guide, you may remember that the best place to modify your configuration is in /etc/fail2ban/jail.local. Since ssh filter is already enabled in the default configuration, you only have to specify the new port:


Same for other services:


Tuesday, 21 December 2010

Installing Fail2ban: an other step in securing your server

Fail2ban is an authentication failure monitor that will check your different logs and, through iptables, can ban hackers and other wrong does from accessing your server.

On Debian (lenny), the installation is quite straight forward:

sudo aptitude install fail2ban

There will be 2 configuration files that you may need to take a look at:
/etc/fail2ban/jail.conf and /etc/fail2ban/jail.local. The first one has good defaults to begin securing your server. Since it can be overwritten when you update fail2ban, you should put your modifications or additions into jail.local.

Here are some things you may want to put in your local version:

Enable checks for services you use, like this for apache:
enabled = true

Fail2ban will log it's action to a log file but I'm not sure I'm going to take a look at them regularly but I do know that I check my e-mails many times a day, so why not have it email you it's logs:
action = %(action_mwl)s

In the same section [DEFAULT], you want to enter the IPs that should never be banned to make sure it won't lock you out of your own server:
ignoreip = xx.xx.xx.xx

I find default ban time a bit short (10 minutes), so once everything seems to work fine, you may when to update that to a longer period:
bantime = 3600

The latest version of fail2ban comes with a bunch of default filters for different services and they are pretty good out of the box. But I encourage you to take a regular look at your logs and see if there is some attacks that fail2ban doesn't catch and try to write your own filters. Here is one example I use to ban people who try to access DB/admin/system stuff through apache, some will block attemps at WIndows vulnaribilities but that's good since those people may then move on to Linux specific ones.

In /etc/fail2ban/jail.local:
enabled = true
port = http,https
filter = web-exploits
logpath = /var/log/apache2/error.log
maxretry = 4

In /etc/fail2ban/filter.d/web-exploits.conf:

failregex = [[]client <HOST>[]] File does not exist:
ignoreregex = [[]client <HOST>[]] File does not exist: .*favicon\.ico

This will ban anyone who tries to access an inexistent resource on my server. Looking at my logs, I find that the favicon one is the only reasonable exception so I added a ignoreregex entry.

As you see the rules are made up of regular expressions, so it's pretty easy to write your own rules.

 Just make sure to restart it when you have made changes:
sudo /etc/init.d/fail2ban restart


Tuesday, 7 December 2010

Installing, Configuring and Using Tripwire (or Securing your Debian Server)

Multiple things must be done to effectively secure your Linux server and using Tripwire is one of them. Tripwire basically checks all important files on your system and lets you know if they change. It's a good tool in theory but to be useful you must learn to configure it and use it properly.

Installation (the easy part):

sudo aptitude install tripwire

Voilà! That's it, just follow the on-screen menus. You'll need to enter (and remember) two password, the local and the site passphrases. For added security, use two different ones and as long and complexe as possible. Why not use phrases that are easy to remember like "Bi2clycle0ridesare4fun!"

Configuration (the important part):

The beauty with Debian packages is that everything is installed and pre-configured so it works out of the box (so to speak). For Tripwire, a default configuration is installed and (if anacron is also installed) a cron job will mail it's report daily to the root user. But you'll find that some files that Tripwire monitors don't exist or change daily (or more often) even on a perfectly sane system. If you keep it like that you'll just get used to seeing errors and won't notice the important one.

You'll find the configuration file at /etc/tripwire/twpol.txt. If it's not there, just create it:

sudo -i
/usr/sbin/twadmin -m p > /etc/tripwire/twpol.txt

compare it with the output of:

sudo tripwire --check

Comment out files in twpol.txt that --check reports has non-existant or directories that you know will always change (like /var/log). Once you're satisfied, run:

sudo /usr/sbin/twadmin -m P /etc/tripwire/twpol.txt
sudo /usr/sbin/tripwire -m i

to load the new policy.

Redo the --check and the configuration until you're satisfied (you should get down to 0 errors reported). For safety measures, you can now delete the twpo.txt file.

Using (the essential part):

OK, Tripwire is installed and configure properly, it monitors your important files/directories and mails you reports every day. Eventually, files will change and Tripwire will report errors. Most of the time those will be known and accepted changes and we must tell Tripwire that we accept those changes or he will continue to report those errors forever.

First, take a look at the report you received or run (and read):

sudo tripwire --check

If everything is OK, run the following command:

sudo tripwire -m u -Z low -r /var/lib/tripwire/report/hostname-timestamp.twr

The hostname will be the one that is return by the hostname command and the timestamp should have the form: yyyymmdd-hhmmss (ex. Choose the latest one, the one that was just created when you ran the --check command. This will open a file where you can accept or refuse the changes. If you have already checked the report and everything is OK, just quit (:quit), enter your local passphrase and Tripwire will re-initialize it's database and be ready to report any new changes from now on (but not report the old ones).

So remember:
- Install Tripwire
- Configure it properly
- Make sure it mails you a daily report
- Read it (daily)
- If all is OK, re-initialize the database.

Monday, 29 November 2010

Installing svn (over SSH) on Debian and OS X

This a quick Howto for installing a Subversion server on Debian/Ubuntu and OS X client to use it. My goal is to use svn over ssh for added security (communication is encrypted and I don't need to open another port).

I assume you have a SSH server already installed and running, plus root access. The svn clients will need an account on the server and read/write access to the svn repository (we'll see how to do that below).

(Don't enter the Debian$ part, it represents the prompt)

Server side
Installing svn:
Debian$ sudo aptitude install subversion
[ Side track:
If you don't do regular maintenance on your server, why don't you do it now:
Debian$ sudo aptitude update
Debian$ sudo aptitude safe-upgrade

Check if you need to restart any deamons/services to use the latest updates:
Debian$ sudo checkrestart
Check out this post!

Preparing the repository:
Debian$ sudo mkdir /var/svn-repos
Debian$ sudo svnadmin create --fs-type fsfs /var/svn-repos/myproject1

  • You don't have to use "svn-repos" or the /var directory.
  • You'll execute the last command for every project you have.

Debian$ sudo groupadd subversion
Debian$ sudo addgroup user1 subversion (repeat for every user, you want to have access)
Debian$ sudo chown -R root:subversion /var/svn-repos/*
Debian$ sudo chmod -R 770 /var/svn-repos/*

At this time, if you're connected through ssh as user1, you may need to reconnect to make that groupadd command effective for your session.

That's it, you'll now be able to access your repository in svn+ssh:// form. Let's see how do do that on OS X (mostly the same for all other OS):

Client side:
Open a Terminal window and run:
Mac$ svn
If the command is not found, you'll need to install it. (The openCollabNet package is probably the easiest way, you'll need to register but you want to keep your email private you can always use services like to do so).

Now that svn is installed on your Mac, let's try to populate our project:
Mac$ mkdir svn
Mac$ cd svn
Mac$ mkdir trunk
Mac$ mkdir branches
Mac$ mkdir tags
Mac$ echo "test only" > trunk/test.txt
Mac$ svn import . svn+ssh:// -m "Test import"

Now unless your Mac username is the same as your Debian account name, and you are using the default ssh port the connection won't work. You'll need to tell svn how to connect with ssh:
Mac$ nano ~/.subversion/config
Go to the [tunnels] section (Ctrl-W in nano) and add the line:
myssh = ssh -p 55555 -l debian_username
Replace 55555 with your own sshd server port and debian_username with your own login name on the server.
(Note: If you never connect to other svn server through ssh, you can use ssh instead of myssh)
Save and exit (Ctrl-X, y).
Let's try again:
Mac$ svn import . svn+myssh:// -m "Test import"
You will be asked for your debian password and you should see the list of files/folders being imported into svn.

You may need to enter svn commands often or one command may generate multiple connections and ssh will ask for your password each time. You can use ssh-agent to provide automatically provide ssh with your credentials:

On your Debian server, check that ~/.ssh exists and if not:
Debian$ mkdir ~/.ssh

Back in OS X:
Create your public/private keys (by default, it will use RSA, 2048-bit encryption):
Mac$ ssh-keygen
(Just follow the prompt, choose the defaults. You don't have to choose a passphrase, but if you do make it greater than 3 chars. You will have to enter your passphrase each time you start ssh-agent. I prefer to have one.)
Copy your public key over to the server:
Mac$ cat ~/.ssh/ | ssh -p 55555 "cat - >> ~/.ssh/authorized_keys"
Test it:
Mac$ ssh -p 55555
In OS X or other X environments a login box should be displayed asking for your passphrase.

ssh-agent can now use your key to log on automatically when it needs to. To go back to our example:
Start ssh-agent like this:
Mac$ eval `ssh-agent -s`
Mac$ ssh-add
(It will ask for your passphrase)
Check out your code:
Mac$ svn co svn+myssh:// .

You can add the ssh-* commands to your .profile, but since I don't connect to svn that often I have put it a separate file that I can just source.
Add this to your ~/.profile or a separate file:

function start_agent {
     echo "Initialising new SSH agent..."
     /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
     echo succeeded
     chmod 600 "${SSH_ENV}"
     . "${SSH_ENV}" > /dev/null
# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     ps ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {

For code that will work in other environment (like cygwin), see this post.

If you put it in your .profile, you'll need to enter your passphrase the first time you open up a Terminal window in a session. If you chose not to have a passphrase, .profile is the perfect place to put this. If you put it somewhere else (let say ssh_cmd), just do this when your ready to connect:
Mac$ source ssh_cmd
Mac$ svn ...


There is a lot of way to setup svn+ssh access, the steps above were the ones I used based on my own needs. You can also have your users connect through the same account while still keeping track of who's changing what and being able to control access with svnauth, check the offcial documentation here.

I will soon be posting about svn on https which is a better setup for most cases.