AIX TLS FTP Client and vsFTP TLS Server

While trying to get the default AIX ftp client working with a FTP over SSL (FTPS) linux server running vsFTP I ran into a few interesting issues that seemed worth documenting.

$ ftp -s ftp.my.com
Connected to ftp.my.com.
220-| Welcome to XXXX FTP over SSL (FTPS) Server |
220-| We allow TLS connections on ports 21 and 990.    |
220-| If you see "503 Login with USER first." use SSL. |
220
234 Proceed with negotiation.
TLS Auth Entered.
Error with certificate at depth 2
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
...

...

...

    Signature Algorithm: sha1WithRSAEncryption
        13...:ff
Error error 20:unable to get local issuer certificate

ERROR  Error during the hand shake for the control connection
ERROR  Error setting BIO object for the control connection
FTP: Unable to authenticate to Server.


First we should check to see if our vsFTP server is handing out the correct portions of the certificate that may include chained certs
1) Private Key
2) Public Key
3) Primary CA chain cert
4) Secondary CA chain cert
5) etc.


This document provides a good idea of howto take a better look at what your server is doing using the openssl client utility.

http://langui.sh/2009/03/14/checking-a-remote-certificate-chain-with-openssl/

$ openssl s_client -showcerts -connect myftp.example.com:21


There is a typo in IBM's documentation on page 371 we see section "6.8.2 Setting up ftp to use TLS"

http://www.redbooks.ibm.com/redbooks/pdfs/sg247430.pdf

This document describes using the "~/.ftpcnt" file to store the path to the CA certificate.  However after running "truss" on the ftp process while trying to connect we find:


kwrite(1, " T L S   A u t h   E n t".., 18)     = 1864 EOPNOTSUPP0) = 0xD0538128
kopen("/home/nbausch/.ftpcnf", O_RDONLY)        Err#2  ENOENT


So, the real file we should use is "~/.ftpcnf".  To get this working you should download the CA bundle for your certificate that your server is presenting.  If you are unsure you can look at the results of your "openssl s_client -showcerts" command above or choose to download an bundle of commonly used SSL certificates like those distributed with curl.

$ wget  http://curl.haxx.se/ca/cacert.pem

Finally we need to create our "~/.ftpcnf" file with the path the our CA certificate file.

$ cat /home/nbausch/.ftpcnf
CA_PATH         /home/nbausch/cacert.pem

And finally we can test our AIX ftp client using TLS to connect to a host:


$ ftp -s ftp.my.com
Connected to ftp.my.com.
220-| Welcome to xxxx FTP over SSL (FTPS) Server |
220-| We allow TLS connections on ports 21 and 990.    |
220-| If you see "503 Login with USER first." use SSL. |
220
234 Proceed with negotiation.
TLS Auth Entered.
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            49:xxxxxx:0e
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=Terms of use at https://www.verisign.com/rpa (c)10, CN=VeriSign Class 3 Secure Server CA - G3
        Validity
            Not Before: Dec 29 00:00:00 2011 GMT
            Not After : Dec 28 23:59:59 2013 GMT
        Subject: C=US, ST=Illinois, L=xxx, O=xxx, OU=xxx, CN=ftp.my.com
TLSv1/SSLv3 ( DES-CBC3-SHA ), 168 bits
Name (ftp.my.com:nbausch): nbausch
331 Please specify the password.
Password:
230- Welcome XXX FTP Service.
230-
230- Notice: Files uploaded to this system will only be kept for
230-         7 days.  Files are subject to anti-virus scanning
230-         and deletion before 7 days if deemed a risk.
230-
230- Known Good FTP Clients & Examples:
230-   FileZilla: (Windows/Mac/Linux) Server Type: (FTPES) Ports: (21/990)
230-   lftp: lftp -u <user>,<pass> ftp.my.com -e "set ftp:ssl-protect-data true; get README.first"
230-   lftp: lftp -u <user>,<pass> ftp.my.com -e "set ftp:ssl-protect-data true; cat README.first"
230-   curl: curl -u <user>:<pass> -k --ftp-ssl-reqd ftp://ftp.my.com/README.first
230-   openssl s_client -starttls ftp -connect ftp.my.com:21
230 Login successful.
200 PBSZ set to 0.
200 PROT now Private.
ftp> passive
Passive mode on.
227 Entering Passive Mode (10,236,32,91,84,157)
150 Here comes the directory listing.
TLSv1/SSLv3 ( DES-CBC3-SHA ), 168 bits
README.first
226 Directory send OK.

32-bit Puppet RPM's & SRPM's For AIX - Howto Build

Background:

This document will cover some suggestions for how to get puppet running on an AIX system.  We will create an environment on AIX 6.1 TL6 that will allow us to compile and or package in RPM zlib, openssl, ruby, facter, and finally puppet.   This recipe yielded working packages on my test systems but your mileage my vary.  This is my first adventure into the AIX world, so I really can't guarantee that these RPMS will work anywhere else, but I hope they do.  If nothing else I hope someone can benefit from the SPEC files and SRPM's for their own environment.

I was unable to find much information on howto get puppet running on AIX, so hopefully this helps someone who is interested in running the puppet agent on AIX and was having similar struggles getting puppet working.

My goal was to try to separate puppet from the rest of the filesystem with all application dependencies (where possible and practical) living in /opt/puppet except for the configuration files which would live in /etc/puppet.  I was only interested in running the puppet agent and had no interest in an AIX pupetmaster.

I also wanted to stick with GCC, and much thanks goes to Michael Perzl for his excellent documents about how-to setup a sane compiler environment for GCC on AIX.  And also thanks to the man for maintaining a diverse set of GNU and open source packages on AIX.



For the impatient:

  32-bit RPM's Compiled on AIX6.1

• pup-facter-1.6.3-1.puppet.local.aix6.1.noarch.rpm
• pup-openssl-1.0.0e-2.32.puppet.local.aix6.1.ppc.rpm
• pup-puppet-2.7.6-1.local.aix6.1.ppc.rpm
• pup-puppet-conf-0.1-1.local.aix6.1.noarch.rpm
• pup-ruby-1.8.7-p352.1.32.puppet.local.aix6.1.ppc.rpm
• pup-zlib-1.2.5-1.32.puppet.local.aix6.1.ppc.rpm

  32-bit SRPM's Used To Compile The Above RPM's

• pup-facter-1.6.3-1.puppet.local.src.rpm
• pup-openssl-1.0.0e-2.32.puppet.local.src.rpm
• pup-puppet-2.7.6-1.local.src.rpm
• pup-puppet-conf-0.1-1.local.src.rpm
• pup-ruby-1.8.7-p352.1.32.puppet.local.src.rpm
• pup-zlib-1.2.5-1.32.puppet.local.src.rpm

Prerequisite Packages and Software:


Here is the OS level I started out with:

$ oslevel -r

6100-06

$ oslevel -s

6100-06-06-1140

We need to install the libm package:

$ lslpp -l | grep -i bos.adt.libm

 bos.adt.libm               6.1.6.0  APPLIED    Base Application Development

We also need to install this APAR or bring your system up to AIX 6.1 TL6 SP6 so we don't suffer from a GCC assembler error:
http://www.perzl.org/aix/index.php?n=Main.GCCAssemblerError

My fresh install of AIX 6.1 had the following RPM's on it;

$ rpm -qa

cdrecord-1.9-7

mkisofs-1.13-4

tcl-8.4.7-3

tk-8.4.7-3

expect-5.42.1-3

TWeagent-7.7.0-0

AIX-rpm-6.1.6.15-2

Next we need to install the following RPM's:

$ rpm -ivh make-3.80-1.aix5.1.ppc.rpm

$ rpm -ivh gcc-4.2.0-3.aix6.1.ppc.rpm

$ rpm -ivh libgcc-4.2.0-3.aix6.1.ppc.rpm
$ rpm -ivh db-3.3.11-4.aix5.1.ppc.rpm

I was able to download the above rpms from IBM here:
http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/ezinstall.html



The Build Environment:

First we setup topdir in our rpmmacros file like normal

$ cat ~/.rpmmacros

%_topdir   /home/nbausc0/rpmbuild

The RPM error messages from the default version of RPM were not very helpful.  If you get an error like this:

$ rpm -ba zlib-1.2.5-1.shc.spec

Unable to open temp file.

Then take a look at your rc environment for RPM with the following:
$ rpm --showrc | grep tmp
-14: _tmppath   %{_var}/tmp


I tried to include the required compiler/linking environment variables in the SRPM spec files that would yield me with a usable package. If you are dealing with another environment and what I have done does not make sense, I encourage you to read the following page from perzl.org and adjust things to meet your needs.
http://www.perzl.org/aix/index.php?n=Main.Instructions#sourcecode


Because of differences in the rpm spec files from Linux environments and the environment we have on AIX it is sometimes nice to extract the final RPM to make sure there are no files going to a location you had not intended.

$ rpm -ivh cpio-2.5-1.aix5.1.ppc.rpm

$ rpm2cpio zlib-1.2.5-1.puppet.local.aix6.1.ppc.rpm | /opt/freeware/bin/cpio -idmv

OpenSSL Linking - rpath and -blibpath

I was curious how others had linked the openssl binaries in the other packages for openssl on AIX and I found this:

$ ldd /usr/bin/openssl

/usr/bin/openssl needs:

         /usr/lib/libc.a(shr.o)

         /usr/lib/libpthreads.a(shr_comm.o)

         /usr/lib/libpthreads.a(shr_xpg5.o)

         /usr/lib/libcrypto.a(libcrypto.so.0.9.8)

         /usr/lib/libssl.a(libssl.so.0.9.8)

         /unix

         /usr/lib/libcrypt.a(shr.o)

$ ldd /opt/freeware/bin/openssl

/opt/freeware/bin/openssl needs:

         /usr/lib/threads/libc.a(shr.o)

         /usr/lib/libpthreads.a(shr_comm.o)

         /usr/lib/libpthreads.a(shr_xpg5.o)

         /opt/freeware/lib/libcrypto.a(libcrypto.so.1.0.0)

         /opt/freeware/lib/libssl.a(libssl.so.1.0.0)

         /unix

         /usr/lib/libcrypt.a(shr.o)

         /usr/lib/threads/libc.a(shr_64.o)

         /usr/lib/libpthreads.a(shr_xpg5_64.o)

         /usr/lib/libcrypt.a(shr_64.o)

 
I experimented with 64-bit and 32-bit builds but ran into an unknown issue somewhere between openssl and ruby so I dropped 64-bit for now.  I learned about "-blibpath:<colon separated path>" which is the Linux equivalent of -rpath to force my application to look for dynamic libraries in a specific directory.  This posting from the postfix mailing list was very helpful in understanding linking differences between linking on AIX and Linux.

The final openssl binary looks like this:

$ ldd /opt/puppet/pup-openssl/bin/openssl
/opt/puppet/pup-openssl/bin/openssl needs:
         /usr/lib/libc.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /usr/lib/libpthreads.a(shr_comm.o)

Ruby Linking - and libgcc_s.a

My goal was to compartmentalise puppet so that changes to the system's openssl/zlib/etc. would not cause any adverse effects on puppet's operation.  I did not however pull libgcc_s.a into my configuration.  Many examples I saw online suggested creating a symlink for libgcc_s.a in /opt/freeware/lib but I decided against this.

To try to keep from changing anything in /opt/freeware, yet still allowing ruby to reference libgcc_s.a I have the pup-openssl RPM create symbolic link where the ruby apps and libraries are already setup to look for libs /opt/puppet/pup-openssl/lib/.


The spec file for pup-openssl does something like this:

$ ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a /opt/puppet/pup-openssl/lib/libgcc_s.a



My thinking here was that if my version of libgcc changed I could correct this by removing/reinstalling the pup-openssl RPM or modifying the symlink.  Maybe this was a really bad idea and I guess time will tell.

Below we can see that ruby's openssl.so library is looking at /opt/puppet/pup-openssl/lib/libgcc_s.a

$ ldd /opt/puppet/pup-puppet/lib/ruby/1.8/rs6000-aix/openssl.so
/opt/puppet/pup-puppet/lib/ruby/1.8/rs6000-aix/openssl.so needs:
         /usr/lib/libdl.a(shr.o)
         /usr/lib/libc.a(shr.o)
         /opt/puppet/pup-openssl/lib/libgcc_s.a(shr.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)

Puppet Configuration File

The pup-puppet package does not install a puppet.conf file. I decided to create my own pup-puppet-conf RPM to distribute my puppet.conf.  Below is what the puppet config file looks like.

$ cat /etc/puppet/puppet.conf
[main]
vardir = /opt/puppet/var
logdir = /opt/puppet/log
rundir = /opt/puppet/run
modulepath = /etc/puppet/modules:/opt/puppet/share/modules
archive_files = true
archive_file_server = archiveserver.local

[agent]
certname = localhost.local
server = puppetmaster.local
report = true
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
graph = true
pluginsync = true

Do You Yearn For The Old Google Days

Does auto-complete drive you nuts and muck with your ability to paste into google's search box.  

Awesome article here:
http://www.google.com/support/forum/p/Web%20Search/thread?tid=5a69f1094357f31b&hl=en

We all can't be as cool as "thelostpineapple", but "bluequoll" seems cool in my book for his very simple solution:
https://www.google.com/webhp?complete=0

Ahh, the pain of actually typing a complete thought returns.

vsFTP MySQL Virtual with Auto-Create Home Directories

Expanding on the howto's shown here:

http://www.cyberciti.biz/tips/centos-redhat-vsftpd-ftp-with-virtual-users.html
http://www.howtoforge.com/pureftpd_mysql_virtual_hosting


We get this error because virtual user directory does not exist.  Attempts at directory auto creation via configuration file fail:

331 Please specify the password.

Password:

500 OOPS: cannot change directory:/opt/ftp/testuser

Login failed.

ftp>

We want MySQL user dir's to be auto-created when a valid user tries to login you will need to change your pam configuration to:

#%PAM-1.0

# Auth in MySQL

auth requisite pam_mysql.so user=vsftpd-ro passwd=readonly host=localhost db=vsftpd table=accounts usercolumn=username passwdcolumn=pass crypt=0

auth required pam_script.so onerr=success dir=/etc/pam-script

# Account in MySQL

account required pam_mysql.so user=vsftpd-ro passwd=readonly host=localhost db=vsftpd table=accounts usercolumn=username passwdcolumn=pass crypt=0

$ cat /etc/pam-script/pam_script_auth

#!/bin/sh

if [ ! -d "/opt/ftp/$PAM_USER" ]; then

  /usr/bin/env mkdir /opt/ftp/$PAM_USER

  /usr/bin/env chown ftp:ftp /opt/ftp/$PAM_USER

fi

Small changes to SPEC file for 64 bit location and packaging:


# see http://www.rpm.org/RPM-HOWTO/build.html for details
# and sub-package details for maximum-rpm too
#
Summary: a PAM module that can invoke scripts within the PAM stack
Name: pam-script
Version: 1.1.4
Release: 1.local
License: GPL
Vendor: LBNL/NERSC
Group: System Environment/Base
Source: http://sourceforge.net/projects/pam-script/pam-script-1.1.4.tar.gz
BuildRoot: /var/tmp/%{name}-buildroot
Provides: pam-script-1.1.4
Requires: pam
%description
pam-script allows you to execute scripts during  authorization,  passwd
changes, and on session opening or closing; which can affect the PAM stack.
----------

%package examples
Summary: some pam-script example scripts and docs
Group: System Environment/Base
%description examples
logscript - simply records the various PAM_* environment variables to
a log file.  Useful for studying how the various PAM modules work together.
tally - a pam-script version of the pam-tally module with much of the
same options.  It's a perl script and is useful for hacking your own
authentication scheme.

%prep
%setup

%build

%pre

%install

# pam-script
./configure    --prefix=/usr                    \
        --libdir=/lib64/security                \
        --sysconfdir=/etc/pam-script            \
        --mandir=/usr/share/man

make DESTDIR=$RPM_BUILD_ROOT install
make DESTDIR=$RPM_BUILD_ROOT install-man7
make DESTDIR=$RPM_BUILD_ROOT install-examples

%post
# pam-script

%preun
# pam-script

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root)
/lib64/security/pam_script.so
%doc README
/etc/pam-script/README
/usr/share/man/man7/pam-script.7.gz
%files examples
/etc/pam-script/*