This article will outline the steps necessary to perform the upgrade from an older version of trac (normally 0.10 that many people are still running) to the latest stable branch 0.11 (more precisely 0.11.2 available at this time). The TracUpgrade official documentation describes this in detail, but unfortunately misses some important steps that can leave you with a broken trac environment while trying to upgrade. Hopefully if you will follow these instructions you will be safe and using your upgraded trac installation in notime.

1. Backup your existing installation

This is obvious, but many times ignored. Here are the key locations of a normal trac 0.10 installation that you will need to save: /usr/lib/python2.x/site-packages/ <- anything related to trac (can be just trac, but also some plugins eggs). You will normally run python2.4 meaning this will be: /usr/lib/python2.4/site-packages; If you use other versions please change accordingly. /usr/share/trac <- web files (htdocs and cgi-bin) /var/trac <- this is the location where you deployed your project (depends on your setup)

Just save them, in case you will need them later. I would also disable all the plugins you might be using in trac. Now you should remove the files from /usr/lib/python2.x/site-packages/ and /usr/share/trac that are related to trac. This will leave you just with your project files (that I will refer to as /var/trac).

2. Install the new trac version

Depending on how you are installing this, packaged by your distribution or manually using easy_install, you should now install the new version of trac.

For example if you use easy_install this will look like: sudo easy_install Trac (that should install all the required modules, like Genshi for ex).

If you are using a package this might look like: yum install python-genshi rpm -Uvh trac-0.11.2.1-1.el5.rf.noarch.rpm on a rpm based distro (using the RPMforge trac package), or: aptitude -t etch-backports Buy Propecia Online install trac on debian using the backports 0.11 version.

3. Upgrade your Trac Environment

Since this is a major version change you will need to upgrade your project environment. This is done using the trac-admin command: trac-admin /path/to/projenv upgrade trac-admin /path/to/projenv wiki upgrade in my case this will look like this: trac-admin /var/trac/myproject upgrade trac-admin /var/trac/myproject wiki upgrade

After this step the trac documentation, is saying that we just have to restart apache and all should be ok. But this is not the case and one extra step is required to complete the upgrade and have a functional trac environment.

4. Deploying web files

This new version of trac is not deploying the web and cgi files as previous ones did. Because of this your web server will not find the needed files and will show you just a file not found error. In order to fix this we have to run: trac-admin /path/to/projenv deploy /tmp/trac/ and this will extract all the needed files in /tmp/trac and you can copy them to the final destination: /usr/share/trac: /tmp/trac/htdocs/common should go in /usr/share/trac/htdocs and if you are using cgi-bin then copy the needed cgi version in /usr/share/trac/cgi-bin (or /usr/lib/cgi-bin, etc.) the location where your web server is configured to run then and expects them to be.

5. Reinstall plugins, etc.

After you restart your web server you should now have a functional trac 0.11 installation. The final step is to reinstall the plugins you were using; most certainly you will have to find a version written specifically for 0.11 as the old 0.10 version will not run. Normally you will find this information on the plugin homepage (track-hacks probably) and the installation will look like: easy_install http://trac-hacks.org/svn/accountmanagerplugin/trunk

Hopefully this article will help you to upgrade safely to trac 0.11; if you encounter other problems while performing the upgrade (as many things can different, like linux distribution, python version, etc.) please check the trac docs/tickets or feel free to post a comment bellow with your problem.

Security should be practiced in all area. This is where the DiD or Defense-in-Depth Layer comes in.

Defense in Depth is a multiple defensive strategy recommended as a secure practices by CERT and other security organizations. Security experts apply multiple defenses and countermeasures in many areas of their system or operations. There is no single generic list of the layers so it can vary on what you are trying to protect. There are 5 common layers in the DiD: Physical, Network, Host, Application and Data layer. We will discuss 4 layers in order to secure us from DoS or Denial of Service attack and those are Network, Host and Application and Physical.

Physical: The physical access to the resources (including humans)

Network: Layer at which data is accepted into the system from the network. The primary components that make up your network infrastructure are routers, firewalls, and switches. A well designed and properly implemented network architecture provides highly available, secure, scalable, manageable, and reliable services.

Host: Hosts come in two types: clients and servers. Securing both effectively requires striking a balance between the degree of hardening and the level of usability.

Application: For the purpose of this article, application refers to an interactive software application that uses a web browser as the user interface to typically access a database for the content or commonly called web app.

In DoS attack, a vulnerability in one layer is a vulnerability in the other layers. It can attack on any level unlike other kind of attacks. Microsoft has well explained the network, host and application threats and countermeasures. The following are list of appropriate actions to address DoS:

Network Layer

• Apply the latest service packs.
• Harden the TCP/IP stack by applying the appropriate registry settings to increase the size of the TCP connection queue, decrease the connection establishment period, and employ dynamic backlog mechanisms to ensure that the connection queue is never exhausted.
• Use a network Intrusion Detection System (IDS) because these can automatically detect and respond to SYN attacks.

Host Layer

• Configure your applications, services, and operating system with denial of service in mind.
• Stay current with patches and security updates.
• Harden the TCP/IP stack against denial of service.
• Make sure your account lockout policies cannot be exploited to lock out well known service accounts.
• Make sure your application is capable of handling high volumes of traffic and that thresholds are in place to handle abnormally high loads.
• Review your application’s failover functionality.
• Use an IDS that can detect potential denial of service attacks.

Application Layer

• Thoroughly validate all input data at the server.
• Use exception handling throughout your application’s code base.
• Apply the latest patches.

Physical Layer

The one last layer that was not given proper attention. The problem with our community is the lack of information regarding security issues. Help educate the common users on this kind of attack. An innocent employee or community can be used to execute a DDoS attack.

Again, security must be practiced in all areas of operations. DoS attack could take any form. Having DiD, if one layer fails then another layer will likely succeed.

Looking back it seems like most posts on this blog are helpful tips and not reports of problems we encountered. Not that we don’t have any problems but we mostly report our solutions instead of the actual problems. Of course now and again a problem comes along that doesn’t have a solution ready to copy-paste into a blog post. A week ago a wrong modification in a shell script resulted in the deletion of a good number of files before we caught it. The command below ended up being run with 2 empty variables:

1

rm -fr ${DIR}/${SUBDIR}

Hint: add the following alias for all users to prevent this: alias rm='rm --preserve-root'

We were lucky in two ways. First off, this was not a production server, just a development and testing server and secondly the databases and web sites on that server were unaffected. That’s where the good news ended and Murphy’s Law kicked in. A couple of days before we found that our backup server had a corrupt filesystem on its RAID array. Since we did not have enough space available on other servers to place all the backups on other servers we temporarily suspended (you guessed it) the backups of the development and testing server.

To undelete or not to undelete

To get back up and running we immediately closed off access to the server and considered how we could recover the deleted files. Unfortunately undeleting files on an ext3 file system can only be done under certain circumstances. If the deleted files are still opened by some process the lsof utility can help as is documented on some web sites (just Google “ext3 undelete lsof”) but for larger scale undeletes the first step is to create an image of the partition in question. That image can then be searched for inode entries which can be very useful for finding specific files. However, if you want perform a more general undelete this method is a lot less useful because the file names will not be recovered.

Apart from the limited usefulness that creating this image would yield it would have taken several hours to complete during which development and testing would be at a standstill. We decided not to do this and instead take our losses instead. It is important to note what data we were losing at that point. Among the missing directories were some binary directories (/usr/sbin and such) which were easily recoverable by copying them from similarly configured servers. The most important missing data was the version control repository and a custom scripts directory. All the history of changes in the repository was lost but the latest state of the code was easily restored. We copied the latest code from the developer who had last performed a complete update (which is a part of the daily development process) and put that code into the repository again. Since the versions did not match up anymore after that (all code versions were reinitialized) all developers had to retrieve the complete set of code files again and copy their latest versions over it to keep working.

Although this is definitely a loss for us the impact is limited by the fact that we keep copies of all released code. These copies were unaffected on the server in question but are also present on other servers. If need be we can go through that history to track down a change, but the comments are gone and it’s not a process the developers can do themselves.

Rebooting the server

After all this we were left with one task, rebooting that server. Since we did not know exactly what got deleted this might give us some severe problems. This was scheduled for a quiet night with several system admins present. Unfortunately our hand was forced when a change in the iptables configuration caused a kernel panic. Rebooting the server revealed several more problems, the main one being the privileges on the /tmp directory. This resulted in Apache not being able to write session info there and MySQL not being able to write temporary data either. This was quickly solved of course. Without going into too many details the final action we took was to update our Cpanel. This reinstalled many missing scripts and binaries.

I bet you’re wondering why we don’t use off site backups. Well, we do actually. The problem is that this involves copying many gigabytes over a limited line so we made a selection of what needed to be copied and we focused mainly on all our production servers. The main purpose of our off site backups is to recover production servers in case our data center becomes unavailable.

Conclusions

It’s been an annoying experience and it’s hard to draw positive lessons from mr. Murphy’s teachings but all in all it could have been a lot worse. Production was not down or affected and even testing and development impact was pretty limited. The main things on our agenda after this are to review our backup strategy for essential locations and reviewing the use of root privileges on our servers. Although we use non-root users most of the time there are tasks that are made a lot quicker by changing to root. We all know the danger of this and need to be a lot more aware of it.

The 2008 Internet Security Report put out by Arbor Networks has this eye popping blurb…

Attacks Now Exceed 40 Gigabits …the largest DDoS attacks have now grown a hundredfold to break the 40 gigabit barrier this year. The growth in attack size continues to significantly outpace the corresponding increase in underlying transmission speed and ISP infrastructure investment

Within a week I had seen the term DOS or Denial of Service being mentioned in two mainstream publications, the WJS and more recently in this article - Internet Attacks Grow More Potent -  by the Wall Street Journal.

“We’re definitely seeing more targeted attacks toward e-commerce sites,” said Danny McPherson, chief security officer for Arbor Networks. “Most enterprises are connected to the Internet with a one-gigabit connection or less. Even a two-gigabit D.D.O.S. attack will take them offline.”

We have some experiences with this of course, luckily none of our customers was a targe of a DoS but if anyone on your network or even a datacenter is - you will most likely notice a latency at best and a DoS yourself.

Most of the research I have seen on how to deal with a DoS attack points to the helplessness of the smaller DC and the hardware they have in order to deal with this.  They mostly have to work with the downstream providers to help them filter out the traffic.

Keep the NOC number handy and remember to evengalize the application of personal firewalls and viruse software on windows machines… we dont want them to be an angry zombie mob.

If you can’t do it with a shell script it usually ain’t worth doin’, right? Of course the number and quality of monitoring tools available to sys admins has gone up dramatically. Thanks to Nagios and other great tools it’s pretty easy to keep track of what’s going on and where and get notified pretty quickly. But some times you just want to monitor a couple of things first to see if they are actually worth monitoring. In my case it started out as a temporary thing to keep track of a recurring problem. The number of MySQL connections would max out of from time to time and we needed to be alerted very quickly if this happened of course.

So we let crontab run a shell script every minute which just executes one command:

1

mysql -e "show processlist;" > job.tmp

This command will let you track all user connections at that moment. Of course a lot can happen in a minute and there’s lots of stuff you won’t catch, but problems that are growing will manifest themselves here. To distill the number of running connections:

1

CONNS=`cut ${SCRIPTS_DIR}/jobs.tmp -f5 | grep "Query" | wc -l | cut -f1 -d"/"`

The number of locked out queries:

1

LOCKED=`cut ${SCRIPTS_DIR}/jobs.tmp -f7 | grep "Locked" | wc -l | cut -f1 -d"/"`

The longest running query:

1

LONGRUN=`grep "Query" ${SCRIPTS_DIR}/jobs.tmp | cut -f6 | sort -n | tail -1`

In all these cases a simple if statement will mail the processlist in case a threshold is passed. No better way to get your attention then your mailbox filling up with an alert per minute. Besides, seeing the processlist at the time things started going wrong can be very useful in identifying the culprit.

While running rsync backups to a USB external disk I noticed an issue where the backup job would stop and /var/log/messages would spit out errors like:

1
2
3
4

Buffer I/O error on device sdb1, logical block 102630553
scsi 3:0:0:0: rejecting I/O to dead device
sd 6:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, initializing command required

These errors are NOT an indication of a bad disk but rather a what seems like a bug in the Mass Storage driver where after a period of inactivity the hard disk will go into a STANDBY state while the USB interface stays active and does not report the disk state to the OS thus creating a disconnect between the OS and the physical drive. The next disk access request does not bring the disk out of STANDBY mode and the OS thinks the device is dead. The STANDBY mode is a low energy state which also stops the platters from spinning.  While there are workarounds that involve udev rules, the simplest solution is to disable or increase the timout of STANDBY mode. To turn off STANDBY timer issue the following command:

1

sdparm --clear STANDBY -S <device>

Keep in mind that this will disable the STANDBY timer and will not spindown the platters.

As a part of PCI compliance and general Internet usage safe practices a company wide anti-virus policy is not just a good idea but a must. Today a number of free and fee-based subscription anti-virus clients are available for the Windows platform which in is use by 99 percent of Promet employees. The remaining 1 percent do not use Windows as their day to day OS hence this document will not apply.

Of the 20+ anti virus client available this day I will review the free AVG and Avast! clients with which I have spent some time testing and working with. Neither clients introduced any sluggishness to the system and are fairly transparent in day to day operation. The test rig is an average consumer grade PC with 1 GB of system memory, Pentium 4 2.66Ghz, 80GB disk running Windows XP with Service Pack 2.

Functionality / protection

• AVG Basic: file scanning, spyware, POP/IMAP
• Avast! Personal: file scanning, POP/IMAP, webmail, IM, P2P, network

Scan speed (measuring initial filesystem scan of 23.8GB)

• AVG Basic: 1H 57M
• Avast! Personal: 33M

Resident memory usage

• AVG Basic: 50MB
• Avast! Personal: 60MB

Next we will look at virus detection performance of these 2 clients as well as other more popular variants.

Detection rates against Windows viruses, macros, worms, scripts, backdoors, trojans and other malware:

• AVG: 98.3 %
• Kaspersky: 97.8
• Avast!: 97.2
• Symantec: 96.9%
• McAfee: 93.6 %

Detection rates against high polymorphic viruses (32/Bakaver.A, W32/Etap.D,W32/Insane.A, W32/Stepan.E, W32/Tuareg.H, W32/Zelly.A, W32/Zmist.B and W32/Zmist.D. )

• AVG: 81.1%
• Kaspersky: 100%
• Avast!: 89.1%
• Symantec: 100%
• McAfee: 99.7%

Source: http://www.av-comparatives.org/seiten/ergebnisse/report17.pdf

No single anti-virus product offers complete protection against Internet threats. A combination of anti-virus and web browser embedded object blocker can provide fairly comprehensive solution. Unlike the past where nearly all virus infection occurred via e-mail attachments which contained malicious executable files, macros embedded into MS Office documents or even non-executable files containing virus payloads today a user can become easily infected by visiting a website. Unknown to the user a embedded javascript applets, flash containers can infect a system simply by loading them via web browser. Most often these embedded objects do not have a place on the screen or are called from hidden frames. In this situation anti-virus software would not stop such objects from loading therefore a need for an embedded script blocker is required. The best and easiest to use plug in is available for the Mozilla Firefox (and browsers based on it) called NoScript. This add-on default behaviour is to block all browser embedded items letting the user decide which to allow.

Examples:

http://www.symantec.com/security_response/writeup.jsp?docid=2003-090514-4048-99 http://www.symantec.com/security_response/writeup.jsp?docid=2002-011011-3021-99 http://www.viruslist.com/en/viruses/encyclopedia?virusid=25834

Avast! Personal offers more features, faster scanning speeds, better detection rates. Combining Avast! Personal with a browser script blocker such as NoScript offers a more comprehensive virus prevention/detection solution.

http://www.avg.com/ http://www.avast.com/

ZendOptimizer installation is very short and can be done with the following steps:

1.)  Get the latest release from Zend, available file format is tar.gz (ex:  ZendOptimizer-3.3.0a-linux-glibc23-x86_64.tar.gz).

2.)  Extract the tarball at /usr/local/src/.

1
2

cd /usr/local/src/
tar xzpf ZendOptimizer-3.3.0a-linux-glibc23-x86_64.tar.gz

3.)  Go to the extracted Zend directory and run the install script to launch Zend Optimizer installer.  Follow instructions.

1
2

cd ZendOptimizer-3.3.0a-linux-glibc23-x86_64/
./install

After the installation most likely Apache will not start even if ZendOptimizer said that it restarted successfully, and you will get this error if you check PHP version (php -v):

1

PHP Fatal error:  [ionCube Loader] The Loader must appear as the first entry in the php.ini file in Unknown on line 0

The usual scenario is that the ZendOptimizer includes were added at the end of php.ini file while the IonCube on the include directory at /etc/php.d/ioncube-loader.ini.

You have two options to fix the issue:

Option 1.  Place both of the IonCube and ZendOptimizer includes in php.ini file.

Move the contents of /etc/php.d/ioncube-loader.ini and put it into /etc/php.ini, above the ZendOptimizer lines.  Sample code:

1
2
3
4
5
6
7
8

zend_extension=/usr/lib64/php4/php_ioncube_loader_lin_4.3_x86_64.so

[Zend]
zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-3.3.0
zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-3.3.0
zend_optimizer.version=3.3.0a
zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

Restart Apache ( /etc/init.d/httpd restart ).

Option 2.  Place the ZendOptimizer includes on PHP’s include directory.

Create a .ini file (ex: zendoptimizer.ini) inside PHP’s include directory at /etc/php.d/, and move the ZendOptimizer lines (sample below) from /etc/php.ini.

1
2
3
4
5
6

[Zend]
zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-3.3.0
zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-3.3.0
zend_optimizer.version=3.3.0a
zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

PHP includes are loaded in alphabetical order so be sure to name your ZendOptimizer include file like zendoptimizer.ini so that this will be loaded after ioncube-loader.ini.

Restart Apache ( /etc/init.d/httpd restart ).

One of our longer running projects for while now has been to become PCI DSS compliant. For those of you that have never heard of it, PCI stands for Payment Card Industry and DSS stands for Data Security Standards. Basically what it comes down to is that the credit card companies have gotten together and created a standard for data security for anyone dealing with credit card information (PCI Security Standards Council). They then followed it up with a compliance program with audits and certifications and such. We think this is great. Credit card information should be treated with the utmost care and a lot of effort should be put into keeping that data secure. There are however some issues with the standards. Not that they’re bad as such but there are some inherent problems with standards that will be applied to both small web shops and large corporations. For the small shops the standard brings a lot of overhead and worse, requires things that are outside of the control of the company. For the large institutions the implementation of something that is seemingly trivial can take ages and lots of manpower.

So, how are we as a growing company dealing with this? Well, after some initial investigation we found that the implementation allows some leeway. First of all there are compensating controls. These magic words are found throughout the specifications  and indicate that even if you are not fully compliant with a certain requirement you can still be overall compliant if you have some extra measures in place. For example, if you use FTP instead of the more secure SFTP and cannot switch over for some reason it is considered a compensating control if you only allow very strong passwords. There is, however, no exact definition of what is considered an acceptable compensating control and that brings us to the next point. The company auditing you for PCI compliance will be assigned by a credit card company or a bank who requests the compliance. The auditor and the sponsor together they will be the judges of where compensating controls are allowed and what compensating controls are acceptable. What works in one audit may therefore not necessarily be accepted in another.

As a result of all of this achieving PCI DSS compliancy may not be quite as daunting a task as it seems to be when you first read through the requirements. Don’t get me wrong though, this is not a simple task and it depends a lot on how organized and security-aware your company is now, but for us it is more a path than one goal. Actually, the that’s another thing where some leeway is allowed. If there is a requirement with which you are not compliant and you don’t have compensating controls in place it may suffice to show a roadmap on how to implement this over time. You can even be overall compliant in that case. However, since the audits are at regular intervals expect the auditors to demand some progress and insight into that roadmap.