I’m sure almost everyone is aware at this point on the failure that caused Amazon cloud computing solution AWS to be down yesterday (and still is at some smaller extend happening at this time as we can see on their status page) for most of the day. This has affected a few big (Reddit, Foursquare, Quora, Heroku, Engine Yard, for ex.) and many small sites hosted in the us-east-1 AWS region. This happened regardless on the availability zone you were in the region US-EAST (this is the oldest one and still the default for many client tools) and questioned the independence and isolation of the availability zones in the AWS infrastructure design.

The failure was specifically related to the EBS drives that made customer instances non responding, but also prevented them to start or stop new instances with the same EBS volumes (that probably 99% have tried immediately as they got paged). There are some sites that had failover mechanisms, but if they were in the same availability zone it was useless (something that looked like a good solution and fast and cost effective). Others, many startups, found out that they had no such mechanism at all, and that they depended way too much on the Amazon reliability. Until this issue, Amazon had a great uptime record; there were many issues but with individual instances, but not such a global issue. You would expect people running their application in the cloud to expect failures and be prepared and I’m sure most of them are compared with applications deployed in the regular datacenter, but apparently there is still much work to be done.

Overall I believe this showed (if we needed a reminder), that failures can happen and anyone can suffer from such a problem (Google had problems, Facebook the same, and Twitter is most of the time down, and now was just Amazon’s turn). We need to be prepared and build and architect our applications with this in mind and be ready to failover. A great example of this is the twilio application design: http://www.twilio.com/engineering/2011/04/22/why-twilio-wasnt-affected-by-todays-aws-issues/

Setup ssh key authentication for password-less login between servers.  For use by ssh/sftp users or scripts.

Source Server (or local system)

Generate RSA key for user on this system, you can also use DSA.  This asks for key pass-phrase but you can leave it blank.

1

ssh-keygen -t rsa

This asks for location to place the generated key, by default it will be your home directory (ex: /home/your_username/.ssh/).  This generates two files:  id_rsa and id_rsa.pub.  Content of id_rsa.pub is what we need to copy to destination server.

Destination Server (or remote server)

Check if you have the directory .ssh on your home (ex: /home/username/.ssh/), if not, create that directory.

1
2

ls  ~/.ssh
mkdir  ~/.ssh

Check if you have existing file authorized_keys on your .ssh directory, if not create it.

1
2

ls  ~/.ssh/ authorized_keys
touch   ~/.ssh/ authorized_keys

Copy content of id_rsa.pub that you created from your source/local server, or execute this command from your source/local server:

1

scp  ~/.ssh/id_rsa.pub username@remote_host:~/.ssh/authorized_keys

Test your password-less login from source to destination server.

While people are still waiting to for Centos 6 (after many month since RHEL6 was released) we got a little teaser in the release of Centos 5.6 last week. This is a minor security release, but it also brings in some interesting additions like ext4 support and php53 packages among others.

If you are running any Centos 5.x minor version you should be able to upgrade just by running:

1

yum update

We already upgraded to Centos 5.6 all our servers and you should do the same also, and hopefully soon we will see the Centos6 release that everyone is waiting for.

For the full list of packages changed/added please see the centos5.6 release notes: http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.6

Setup sftp-only access on default Redhat/CentOS install. Use built-in sftp-server which is non-chrooted and easy to setup.

Server: Locate sftp-server binary and add to your list of valid shells on the system.

1

which sftp-server

Note:   This will most likely give you this: /usr/libexec/openssh/sftp-server

1

echo /usr/libexec/openssh/sftp-server >> /etc/shells

Note:  You may want to backup your /etc/shells first

User Accounts:  Add or modify accounts to use sftp-server.

New User

1

useradd -s /usr/libexec/openssh/sftp-server  sftponlyuser

Note:  Important parameter there is the “-s /usr/libexec/openssh/sftp-server”, which sets the default shell for this account.   By default you will have “/bin/bash”.

Existing User:

1

vi /etc/passwd

Find the user that you want to configure for sftp only access and edit its shell.  For exampleyou will see a line “username:x:500:500::/home/username:/bin/bash” _ change ”/bin/bash” to ”/usr/libexec/openssh/sftp-server_” and save.  Note: _ Y_ou may want to backup your /etc/passwd file as well.

The common headache for beginners who try to move their Drupal site with CiviCRM is the CiviCRM configuration on new location or domain. I did this countless times and dealt with different problems before. I’ve read many online tutorials and how-tos, and I finally have my notes which is working for me everytime i move Drupal and CiviCRM site.

Let’s assume you already moved the databases (separate database for Drupal and CiviCRM as a recommended setup) and the files to new location and have set the correct file permissions as well. Here is the part that you need for your CiviCRM to work on new location and domain:

• access your database and empty civicrm.domain:config_backend. You see this anywhere with CiviCRM guide.
• update database details, site path/directory, and domain on civicrm.settings.php
• emtpy sites/default/files/civicrm/template_c (or make this entire dir writable by web user)
• login to you new Drupal site and visit these urls: http://sitename/civicrm/menu/rebuild?reset=1 http://sitename/civicrm/admin/setting/updateConfigBackend?reset=1

Note: This will rebuild the settings for you - if not, repeat/review all the above steps.

I’ve written few Drupal ApacheSolr install guides here and this one is basically an install draft for Ubuntu.

Environment: Ubuntu 10.10, Apache, PHP, MySQL, Drupal 6.20

Things to install / setup: Solr, Tomcat6, ApacheSolr module for Drupal, and SolrPHPClient library

1.) Install Tomcat and setup your Tomcat admin user.

1
2
3
4
5
6

aptitude install tomcat6 tomcat6-admin tomcat6-common tomcat6-user
vi /etc/tomcat6/tomcat-users.xml
  <role rolename="admin"/>
  <role rolename="manager"/>
  <user username="tomcat" password="password" roles="admin,manager"/>
/etc/init.d/tomcat6 restart

If all is good you will be able to access Tomcat admin at http://hostname:8080. Default page will show with links to admin section, etc.

2.) Install Solr

1

wget http://apache.rediris.es/lucene/solr/1.4.1/apache-solr-1.4.1.zip

check for updates

1
2
3
4
5
6
7
8
9
10

unzip apache-solr-1.4.1.zip
mkdir /usr/share/tomcat6/webapps
cp apache-solr-1.4.1/dist/apache-solr-1.4.1.war /usr/share/tomcat6/webapps/solr.war
cp -r apache-solr-1.4.1/example/solr /usr/share/tomcat6/solr
vi /etc/tomcat6/Catalina/localhost/solr.xml
  <Context docBase="/usr/share/tomcat6/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
  <Environment name="solr/home" type="java.lang.String" value="/usr/share/tomcat6/solr" override="true" />
  </Context>
chown -r tomcat6.tomcat6 /var/lib/tomcat6
/etc/init.d/tomcat6 restart

You should see Solr access on your Tomcat admin/manager page (http://hostname:8080/manager/html).

3.) Connect Drupal Site to Solr (multi-core setup)

I already have a Drupal site with ApacheSolr module installed and SolrPHPclient library.

PHP is slow. This should not be a huge revelation for any of us, and this is not because PHP is a bad programing language, or because Python or Ruby are better. This is because it is an interpreted language. Every time you execute a PHP script it is going to do the exact same thing and interpret your code and then execute it over and over. This is obviously slow and doesn’t compare with any compiled language like C++ for example. But we don’t want to write in C++ for our webapps, right? and we would like to use the PHP simplicity as many other open source products and developers; what can we do then? We can very easily enable any of the opcode cache available: APC, eaccelerator or xcache (I’m not aware of something similar for python or ruby, but if it exists you should probably use it also). An opcode cache or PHP accelerator, will cache the compiled form of the php scripts speeding up dramatically their execution time. Many people have done this a long time as they needed to improve the performance of their sites, and get results up to 20x speed up from plain PHP. A very simple step-by-step intro on how you can install APC can be found on this older post.

Now here is the question a good friend of mine asked me a while ago: why is APC not enabled by default? Why do people have to do an extra step to install an external module for this (that many don’t). And my answer at that time was that opcode caches are not perfect, and depending by how you write your code they might cause problems and have it no longer functioning correctly. I think I was quite happy with my answer as myself I would always install an opcode cache on any high traffic site I managed from day one (most of the time APC, but also eaccelerator) and many time even running comparisons between them to see if one of them is faster. Still even today, in most of the performance projects we completed, we found people don’t do this, and they don’t have APC installed and enabled. There are some tricks on how to tune APC and improve even better its performance but here I’m talking just to have it installed. This is why I think that APC should be enabled by default everywhere (not only by ‘high traffic’ sites). It should give PHP scripts a boost of performance and if you really have problems with it you can disable it on a vhost or directory level very simple just by adding in .htaccess: php_flag apc.cache_by_default On

I’m hoping more and more people will do this so we can focus on more serious performance problems in our PHP webapps. I don’t see why not, because this is very simple and there are plenty of howto’s available. Still if you need help, and you would like us to do it for you, please contact us and let us speed up your site.

Note: if you have done this many years ago and want somethign even faster for your PHP code you should take a look at HipHop the project from Facebook to compile your php scripts (actually transforms PHP source code into highly optimized C++ and then uses g++ to compile it to machine code) and run them as executables (but this is going to be much harder to implement, but still a very interesting project to keep an eye).

Besides Gmail, many sites are moving to GoogleApps for hosting their emails, and if you use Gmail or GoogleApps a lot then you might love the keyboard shortcuts feature.

You can quickly see available shortcut keys by typing “?” (question mark, no qoutes) and it will give you this screen.

Of course you need to enable keyboard shortcuts on your settings. :)

Below is my presentation from Drupal Con 2011 where Promet Source, a Drupal Development Company based out of Chicago was a Gold sponsor. Although none or our proposed talks got in, we did get a slot at the Main stage. Below are the slides of my talk. Basically we wanted to highlight that while we do focus on automation, performance and security, with Drupal the security updates must be integrated with human support to ensure that updates do not break the site.

What should you expect from your Drupal Web Host

View more presentations from Promet Solutions

Quick install guide for SVN with mod_dav_svn (http/https) repository access on Cpanel server running CentOs5.

Configure Cpanel for SVN

Recompile Apache/PHP (WHM -> Software -> EasyApache) and enable support for the following:

• Dav (Among other things mod_dav can be used by DAV enabled Frontpage clients instead of FP extensions)
• BerkeleyDB. Note: BerkeleyDB support (”–with-berkeley-db”) is not enabled by default in EasyApache so you need to add this manually to your EasyApache config using this guide or this one.

Install Subversion and mod_dav_svn

Install subversion using yum (yum install subversion) or install it from source. At this time CentOS repo version is 1.4.x and the latest from source is 1.6.x.

Install and enable mod_dav_svn in Cpanel

• Install mod_dav_svn using yum (yum install mod_dav_svn) - version 1.4.x same as subersion above.
• Load mod_dav_svn module to Apache. Go to WHM -> Service Configuration -> Apache Configuration -> Include Editor -> Pre-Main Include. Select either ”All Versions” or your current Apache version (ex: ”2.2.16”) from drop-down list. Or you can edit their corresponding file directly:
  • All Versions: /usr/local/apache/conf/includes/pre_main_global.conf
  • Apache 2.2.16: /usr/local/apache/conf/includes/pre_main_2.conf
• Add the codes below and click ”Update” button to save config. This will restart Apache if no error on your configuration.

1
2
3
4

<IfModule mod_dav.c>
LoadModule dav_svn_module /usr/lib/httpd/modules/mod_dav_svn.so
LoadModule authz_svn_module /usr/lib/httpd/modules/mod_authz_svn.so
</IfModule>

Create repository and add http access (ex: http://domain.com/svn)

• Create your repository: svnadmin create /path/to/svn/repos
• Add the code below to your domain’s custom include file. You need to create this file based on this guide to preserve when you recompile Apache/PHP. Example file (Apache2,standard/non-ssl): /usr/local/apache/conf/userdata/std/2/cpanl_user/domain.com/svn.conf File Contents:

1
2
3
4
5
6
7
8
9
10
11
12
13

<Location /svn>
#  mod dav svn support and location of svn repo files
DAV svn
SVNPath /path/to/svn/repos
# authentication for security, create using htpasswd
AuthType Basic
AuthName "SVN Access"
AuthUserFile /path/to/file/containing/user.pass
Require valid-user
# added for permissions/access
Order allow,deny
Allow from all
</Location>