2011/07/22

Folding@Home in Debian GNU/Linux Squeeze amd64 as a service

I think distributed computing is fascinating, these days I have been learning about MPI, clusters and related stuff. The power of a lot of machines working collaboratively to solve a problem... amazing!, I love computing :-)

You know that a mayor part of time your CPU is not fully used because your processes are not all heavyweight all the time, lots of IO locks, dead times, etc. I mean, rarely ours CPUs are 100% used.


Folding@Home is a distributed computing project where protein folding is studied doing complex simulations, so complex and CPU demanding that the scientists turn to ask for help to the whole world. You can share some cycles of your CPU to the Science.

"Protein folding is linked to disease, such as Alzheimer's, ALS, Huntington's, Parkinson's disease, and many Cancers. Moreover, when proteins do not fold correctly (i.e. "misfold"), there can be serious consequences, including many well known diseases, such as Alzheimer's, Mad Cow (BSE), CJD, ALS, Huntington's, Parkinson's disease, and many Cancers and cancer-related syndromes."

I joined to help anonymously some years ago and I hope you can join us, create a team or just add yourself to the default group.

Check the webpage, they have been awarded and several results have been published.

I have installed the client in my dual boot machines (Windows/Linux) and in my home server.

I will show you how to install FAH in Debian GNU/Linux Lenny in a machine without X and in a service mode. Setting FAH as a service will allow it to start automatically every time the system boots up and you still are in charge because you can disable or even uninstall the service.

Log in as root or sudo as you like.

You should create a new user account by security reasons and according with the best practices suggested here (or here), the best place to put our client is /opt/fah.

# groupadd fah
# useradd -g fah fah --shell /bin/bash -d /opt/fah -m

From here you can download the installer, I chose FAH6.34-Linux64.tgz bacause my linux box is:


uname -a
Linux blogger 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC 2011 x86_64 GNU/Linux

wget http://www.stanford.edu/group/pandegroup/folding/release/FAH6.34-Linux64.tgz
tar xzf FAH6.34-Linux64.tgz
chmod u+x fah6

After that we may configure the client, buy we should do this as the fah user (will create configuration files and data):

su fah 

Then, we run the executable:

$ ./fah6

Note: Please read the license agreement (fah6 -license). Further
use of this software requires that you have read and accepted this agreement.



--- Opening Log file [July 22 00:10:47 UTC]


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.34

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /opt/fah
Executable: ./fah6


[00:10:47] Configuring Folding@Home...

User name [Anonymous]? blogger (any username you like)
Team Number [0]? (The default team, create your team or join any one)
Passkey []?
Ask before fetching/sending work (no/yes) [no]? (this should be no as a service)
Use proxy (yes/no) [no]?
Acceptable size of work assignment and work result packets (bigger units
 may have large memory demands) -- 'small' is <5MB, 'normal' is <10MB, and
 'big' is >10MB (small/normal/big) [normal]?
Change advanced options (yes/no) [no]?

[00:12:18] - Ask before connecting: No
[00:12:18] - User name: blogger (Team 0)
[00:12:18] - User ID not found locally
[00:12:18] + Requesting User ID from server
[00:12:19] - Machine ID: 1
[00:12:19]
[00:12:19] Work directory not found. Creating...
[00:12:19] Could not open work queue, generating new queue...
[00:12:19] - Preparing to get new work unit...
[00:12:19] Cleaning up work directory
[00:12:19] + Attempting to get work packet
[00:12:19] - Connecting to assignment server
[00:12:21] - Successful: assigned to (171.67.108.52).
[00:12:21] + News From Folding@Home: Welcome to Folding@Home
[00:12:21] Loaded queue successfully.
[00:12:26] + Closed connections
[00:12:26]
[00:12:26] + Processing work unit
[00:12:26] Core required: FahCore_78.exe
[00:12:26] Core not found.
[00:12:26] - Core is not present or corrupted.
[00:12:26] - Attempting to download new core...
[00:12:26] + Downloading new core: FahCore_78.exe
[00:12:28] + 10240 bytes downloaded
(many bytes downloaded here...)
[00:12:38] + 1134407 bytes downloaded
[00:12:38] Verifying core Core_78.fah...
[00:12:38] Signature is VALID
[00:12:38]
[00:12:38] Trying to unzip core FahCore_78.exe
[00:12:39] Decompressed FahCore_78.exe (3435296 bytes) successfully
[00:12:39] + Core successfully engaged
[00:12:45]
[00:12:45] + Processing work unit
[00:12:45] Core required: FahCore_78.exe
[00:12:45] Core found.
[00:12:45] Working on queue slot 01 [July 22 00:12:45 UTC]
[00:12:45] + Working ...
[00:12:45]
[00:12:45] *------------------------------*
[00:12:45] Folding@Home Gromacs Core
[00:12:45] Version 1.90 (March 8, 2006)
[00:12:45]
[00:12:45] Preparing to commence simulation
[00:12:45] - Looking at optimizations...
[00:12:45] - Created dyn
[00:12:45] - Files status OK
[00:12:46] - Expanded 374844 -> 1805564 (decompressed 481.6 percent)
[00:12:46] - Starting from initial work packet
[00:12:46]
[00:12:46] Project: 6888 (Run 876, Clone 0, Gen 3)
[00:12:46]
[00:12:46] Assembly optimizations on if available.
[00:12:46] Entering M.D.

  Gromacs is Copyright (c) 1991-2003, University of Groningen, The Netherlands
        This inclusion of Gromacs code in the Folding@Home Core is under
        a special license (see http://folding.stanford.edu/gromacs.html)
         specially granted to Stanford by the copyright holders. If you
          are interested in using Gromacs, visit www.gromacs.org where
                you can download a free version of Gromacs under
         the terms of the GNU General Public License (GPL) as published
       by the Free Software Foundation; either version 2 of the License,
                     or (at your option) any later version.

[00:12:52] Protein: ALZHEIMER DISEASE AMYLOID
[00:12:52]
[00:12:52] Writing local files
[00:13:01] Extra SSE boost OK.
[00:13:01] Writing local files
[00:13:01] Completed 0 out of 250000 steps  (0%)


Ok, here we break the download of unit works [CONTROL+C] because we should setup fah as a service. It will continue from here when the binary be executed again anyway.

As root again, let's create /etc/init.d/fah with this content:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          Service script for Folding@Home.
# Required-Start:    $syslog
# Required-Stop:     $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: This manages the service for client of the distributed computing project Folding@Home.
# Description:      
Service script for Folding@Home.
#
### END INIT INFO

check_config()
{
        if [ ! -e /opt/fah/client.cfg ]; then
                echo "FAH has not been configured!!!"
                exit 1
        fi
}

case "$1" in
        start)
                check_config
                echo -n "Starting FAH..."
                cd /opt/fah
                start-stop-daemon --start --quiet --chdir /opt/fah -b --chuid fah --exec /opt/fah/fah6
                echo "."
        ;;

        stop)
                echo -n "Stopping FAH..."
                start-stop-daemon --stop --quiet --oknodo --user fah
                echo "."                                                                                     
        ;;                                                                                                   
                                                                                                             
        *)                                                                                                   
                echo "Usage: /etc/init.d/fah {start|stop}"                                                   
                exit 1
esac

exit 0

We make it executable:

chmod u+x /etc/init.d/fah

And we setup the service but this is Squeeze so we use insserv instead of update-rc.d (check this).

insserv /etc/init.d/fah

Now, we start our new service for the first time:

invoke-rc.d fah start

We can see what is happening in the background using tail:

$ cd /op/fah
$ ls -ltr
total 3864
-rwx------ 1 fah fah  314840 Feb 23 20:33 fah6
-rw-r--r-- 1 fah fah  146345 Feb 23 20:33 FAH6.34-Linux64.tgz
-rwxr-x--- 1 fah fah     110 Jul 21 20:12 client.cfg
-rw-r--r-- 1 fah fah    5214 Jul 21 20:12 MyFolding.html
-rw-r--r-- 1 fah fah       8 Jul 21 20:12 machinedependent.dat
-rw-r--r-- 1 fah fah    7168 Jul 21 20:12 queue.dat
-rwxr-x--- 1 fah fah 3435296 Jul 21 20:12 FahCore_78.exe
drwxr-x--- 2 fah fah    4096 Jul 21 20:39 work
-rw-r--r-- 1 fah fah     163 Jul 21 20:39 unitinfo.txt
-rw-r--r-- 1 fah fah    8527 Jul 21 20:39 FAHlog.txt
$ tail -f FAHlog.txt
[00:39:10] Project: 6888 (Run 876, Clone 0, Gen 3)
[00:39:10]
[00:39:10] Assembly optimizations on if available.
[00:39:10] Entering M.D.
[00:39:30] (Starting from checkpoint)
[00:39:30] Protein: ALZHEIMER DISEASE AMYLOID
[00:39:30]
[00:39:30] Writing local files
[00:39:39] Completed 2500 out of 250000 steps  (1%)
[00:39:39] Extra SSE boost OK.



(This continues... [CONTROL+C])

And that is all. Top will show you the process a FahCore using 100% of one of your cores but with the highest nice value, that is to say, with the lowest scheduling priority. So, if you don't use the CPU, FAH will do and you will be helping some scientist to find the cure to some ugly illness.























Take a look at this ranking, these papers (78!), these awards or these videos. Finally meet Vijay Pande!

So, Will you join Folding@Home?

2011/07/19

Truecrypt in Debian GNU/Linux Lenny x64

Sometimes you need to encrypt some files like documents, passwords, etc. and puntual encryption maybe just the solution with gpg. However, if you need to encrypt lots of files you should try TrueCrypt for a masive encryption.

So, what about a virtual encrypted partition containing all your files?

The following procedure was done in a remote machine through ssh: 

First, you need to download the installer from here according to your architecture.

uname -a
Linux felix 2.6.32-bpo.5-xen-amd64 #1 SMP Thu Oct 21 13:52:59 UTC 2010 x86_64 GNU/Linux

In my case: truecrypt-7.0a-linux-console-x64.tar.gz

Second, you need some dependencies (?):

apt-get install fuse-utils libdevmapper dmsetup

Third, you must install TrueCrypt:

tar xzf truecrypt-7.0a-linux-console-x64.tar.gz
./truecrypt-7.0a-setup-console-x64

TrueCrypt 7.0a Setup
____________________


Installation options:

 1) Install truecrypt_7.0a_console_amd64.tar.gz
 2) Extract package file truecrypt_7.0a_console_amd64.tar.gz and place it to /tmp

To select, enter 1 or 2: 1

Before you can use, extract, or install TrueCrypt, you must accept the
terms of the TrueCrypt License.

Press Enter to display the license terms...

Press Enter or space bar to see the rest of the license.


TrueCrypt License Version 3.0

Software distributed under this license is distributed on an "AS
IS" BASIS WITHOUT WARRANTIES OF ANY KIND. THE AUTHORS AND
DISTRIBUTORS OF THE SOFTWARE DISCLAIM ANY LIABILITY. ANYONE WHO
USES, COPIES, MODIFIES, OR (RE)DISTRIBUTES ANY PART OF THE
SOFTWARE IS, BY SUCH ACTION(S), ACCEPTING AND AGREEING TO BE
BOUND BY ALL TERMS AND CONDITIONS OF THIS LICENSE. IF YOU DO NOT
ACCEPT THEM, DO NOT USE, COPY, MODIFY, NOR (RE)DISTRIBUTE THE
SOFTWARE, NOR ANY PART(S) THEREOF.


I. Definitions

1. "This Product" means the work (including, but not limited to,
source code, graphics, texts, and accompanying files) made
available under and governed by this version of this license
("License"), as may be indicated by, but is not limited to,
copyright notice(s) attached to or included in the work.

2. "You" means (and "Your" refers to) an individual or a legal
entity (e.g., a non-profit organization, commercial
organization, government agency, etc.) exercising permissions
granted by this License.

3. "Modification" means (and "modify" refers to) any alteration
of This Product, including, but not limited to, addition to or
deletion from the substance or structure of This Product,
translation into another language, repackaging, alteration or
removal of any file included with This Product, and addition of
any new files to This Product.

... ([q] to skip)
..
.

Do you accept and agree to be bound by the license terms? (yes/no): yes

Uninstalling TrueCrypt:
-----------------------

To uninstall TrueCrypt, please run 'truecrypt-uninstall.sh'.

Installing package...
usr/bin/truecrypt
usr/bin/truecrypt-uninstall.sh
usr/share/truecrypt/doc/License.txt
usr/share/truecrypt/doc/TrueCrypt User Guide.pdf

Press Enter to exit...


Four, let's create a virtual disk "Private.tc":


truecrypt Private.tc


Volume type:
 1) Normal
 2) Hidden
Select [1]: 1

Enter volume path: Private.tc

Enter volume size (sizeK/size[M]/sizeG): 1G

Encryption algorithm:
 1) AES
 2) Serpent
 3) Twofish
 4) AES-Twofish
 5) AES-Twofish-Serpent
 6) Serpent-AES
 7) Serpent-Twofish-AES
 8) Twofish-Serpent
Select [1]: 1

Hash algorithm:
 1) RIPEMD-160
 2) SHA-512
 3) Whirlpool
Select [1]: 2

Filesystem:
 1) None
 2) FAT
 3) Linux Ext2
 4) Linux Ext3
 5) Linux Ext4
Select [2]: 4

Enter password: ***********************************
Re-enter password: ***********************************

Enter keyfile path [none]:

Please type at least 320 randomly chosen characters and then press Enter:
Characters remaining: 30 (Press [ENTER] to end or know how much is left)


Done: 100.000%  Speed:   29 MB/s  Left: 0 s

The TrueCrypt volume has been successfully created.


To mount our disk:

truecrypt --mount Private.tc /mnt/truecrypt0

root:/mnt/truecrypt0# ls -ltr
total 16
drwx------ 2 root root 16384 2011-07-19 02:17 lost+found



Now we can move/copy our stuff!


To unmount our disk:

truecrypt -d Private.tc 

References

http://www.howtoforge.com/truecrypt_data_encryption
http://esaucairn-almuric.blogspot.com/2009/03/truecrypt-on-debian.html

2011/07/14

Assembling an executable jar into a timestamped package

I needed to build a package for an executable jar containing all its dependencies, in bzip2 format and with a time stamp in the file-name. This is how a did it:

To generate the time stamp I chose gmaven-plugin but you can use the buildnumber-maven-plugin if you need to relate some build number with your control version system (svn maybe?). Futhermore gmaven is a whole new tool box for a developer!

<plugin>
    <groupId>org.codehaus.gmaven</groupId>
    <artifactId>gmaven-plugin</artifactId>
    <version>1.3</version>
    <executions>
        <execution>
            <id>set-timestamp</id>
            <phase>initialize</phase>
            <goals>
                <goal>execute</goal>
            </goals>
            <configuration>
                <source>
                    def timestamp = new Date().format('yyyyMMdd-HHmmss')
                    project.properties.setProperty('timestamp', timestamp)
                </source>
            </configuration>
        </execution>
    </executions>
</plugin>


To assembly the package the job is done in two parts: first, the jar must be generated with a manifest declaring the main class to execute and the jar must contain all the dependencies needed to run. Second, the compressed file containing the jar and some shell scripts must be named using a time stamp


<plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>2.2.1</version>
    <executions>
        <execution>
            <id>jar-with-dependencies</id>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <addClasspath>true</addClasspath>
                        <mainClass>org.project.bla.bla.bla.ui.cli.Main</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </execution>

        <execution>
            <id>dist</id>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration>
                <finalName>ProjectXYZ-${project.version}-${timestamp}</finalName>
                <descriptors>
                    <descriptor>src/main/assembly/distributable.xml</descriptor>
                </descriptors>
            </configuration>
        </execution>
    </executions>
</plugin>

Here an assembly descriptor comes to help, we include some shell scripts and specify output format:

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
    <id>dist</id>
    <includeBaseDirectory>true</includeBaseDirectory>
    <formats>
        <format>tar.bz2</format>
    </formats>
    <fileSets>
        <fileSet>
            <includes>
                <include>**/**jar-with-dependencies.jar</include>
            </includes>
            <outputDirectory>bin</outputDirectory>
        </fileSet>
        <fileSet>
            <directory>${project.basedir}/misc/shellScripts</directory>
            <fileMode>0755</fileMode>
            <lineEnding>lf</lineEnding>
            <includes>
                <include>*</include>
            </includes>
            <outputDirectory>bin</outputDirectory>
        </fileSet>
    </fileSets>
</assembly>

Of course, the assembly descriptor must located in the place declared:

















This, you can build packages like this: ProjectXYZ/target/ProjectXYZ-1.0-SNAPSHOT-20110714-221041-dist.tar.bz2

mvn package

The jar would be called like this:

java -jar ProjectXYZ-1.0-SNAPSHOT-jar-with-dependencies.jar

What a ugly name, maybe I need a installer or a shell script as a wrapper:

#!/bin/bash
java -Xms300M -Xmx300M -jar $PROJECTXYZ_HOME/ProjectXYZ-1.0-SNAPSHOT-jar-with-dependencies.jar $*

Ok, I need a installer.

Deploying with maven-antrun-plugin

I really like to use maven in my software projects but sometimes it can be a real pain in the "keyboard".

After trying to deploy a package using the maven-deploy-plugin into a ssh/sftp server without success I changed the tactical for old and very known tool: ant

Well, it just worked!

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-antrun-plugin</artifactId>
    <version>1.6</version>
    <dependencies>
        <dependency>
            <groupId>ant</groupId>
            <artifactId>ant-jsch</artifactId>
            <version>1.6.5</version>
        </dependency>
        <dependency>
            <groupId>com.jcraft</groupId>
            <artifactId>jsch</artifactId>
            <version>0.1.44</version>
        </dependency>
    </dependencies>               
    <executions>
        <execution>
            <id>upload</id>
            <phase>deploy</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <target>
                    <scp
                        todir="username:password@hostname:/path/destiny"
                        port="22" trust="true" verbose="true"
                        file="${basedir}/target/ProjectXYZ-${project.version}-dist.tar.bz2">
                    </scp>
                </target>
            </configuration>
        </execution>                   
    </executions>
</plugin>  

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-deploy-plugin</artifactId>
    <version>2.6</version>
    <configuration>
        <skip>true</skip>
    </configuration>

</plugin> 


The maven-antrun-plugin calls the ant task scp to do the transfer to the remote site and needs the declared dependencies.
The trust attribute allows me to accept unknown machines and avoid the exception if it is the case.
The verbose attribute show some progress (useful when the file is big).

The transference must be done every time I release a version of the project and so I attached it to the deploy phase. As I only need the package I deactivated the maven-deploy-plugin using the skip declaration in the configuration.
The file being transferred was build using the maven-assembly-plugin.

mvn deploy

Good, no more manual sftp.