Hudson (Jenkins) and Git HTTP authentication
Posted by Dustin in How-To, Uncategorized on February 13, 2011
I run a Hudson, (now Jenkins) CI server on windows XP.
The Jenkins git plugin is broken if you have repositories that use HTTP auth (See: http://issues.hudson-ci.org/browse/HUDSON-3807, http://issues.hudson-ci.org/browse/HUDSON-7091).
If you try to run a build on a job with a repo using HTTP auth, the process will hang at the repository checkout, as GIT is waiting for a password to be entered. Fortunately, you can set the credentials for the server access in a netrc file placed in your home directory.
Find your home directory:
- If you are running Jenkins as the user you are currently logged in as:
- Commonly, the home directory that GIT will use can be found by going start->Run and typing a “.” in the run dialog (without the quotes). On windows XP this is “C:\Documents and Settings\Administrator”
- To be extra sure, you should open the GIT bash, type “cd” to go back to your home directory, and then enter “pwd” to show the current path.
- If you are running Jenkins as a service
- Create a new Execute Shell build step. Execute the command:
echo $HOME
- Run the build and examine the output to see where $HOME is located
- Create a new Execute Shell build step. Execute the command:
Create the _netrc file:
On linux, this file is actually .netrc, however I found that on windows its using the underscore notation. When creating the _netrc file, ensure that your editor is saving the file using unix-line-endings. If you are using a tool like notepad++ this can be set by going Edit -> EOL Conversion -> UNIX Format.
_netrc contents:
machine <server name>
login <server username>
password <server password>
Where <server name> is the name of the server that hosts your GIT repository, and the username and password correspond to valid HTTP credentials. See the netrc page in references below for additional examples.
Verify the _netrc settings work by attempting to download the HEAD from the repository using curl:
curl --netrc --location -v http://<server_name>/<git_repo>.git/HEAD
This should return something like “ref: refs/heads/master“. If you get an authentication failed message, be sure to scroll up in the console to verify if curl was able to find the server entry in _netrc.
Gotachas:
- If you specify the address to the GIT repository in Hudson in http://<username>@<server> notation, the _netrc data will not be picked up. This is because of the “<username>@” prefix
- I had to modify the git executable I was using, as the default wasn’t picking up the _netrc file
- Under Manage Hudson -> Configure -> Git Installation, point the default executable to the git.cmd script. Mine was at C:\Program Files\Git\cmd
References:
http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt
http://www.mavetju.org/unix/netrc.php
XBMCbuntu Ubuntu 10.04 Howto
For the past couple of years I have been experimenting with a variety of different Media Center distributions for my media PC. I’ve played with Boxee, Moovidia, MythTV, XBMC and LinuxMCE. Since I’m from Canada, Boxee wouldn’t work to its full extent, I don’t have a tuner card, so most of MythTV’s power was lost on me and LinuxMCE seemed like it was dieing at the time. After struggling with the performance (or lack there of) of Moovidia on my old hardware, I eventually settled on XBMC.
I’ve been happy using XMBC for the past year or so and its made some good steps, especially with the release of v. 10 (code name Dharma). It was almost getting to the point where it could play 1080p video without skipping half the frames. Unfortunately I made the mistake of trying to upgrade my distribution to Ubuntu 10.10 (Maverick Meerkat), which apparently breaks my legacy embedded Intel graphics.
Despite several hours of googling I couldn’t get past the errors with the intel driver, namely:
(EE) intel(0): [drm] failed to set drm interface version.
(EE) intel(0): Failed to become DRM master.
DRM_IOCTL_I915_GEM_APERTURE failed: Bad file descriptor
(EE) intel(0): failed to get resources: Bad file descriptor
(EE) intel(0): Kernel modesetting setup failed
So I’ve decided to roll back (AKA: Wipe and re-install) Ubuntu 10.04.
This time around, rather than doing a desktop install of Ubuntu, I’ve decided to do a stream-lined XBMC-Live installation, but with a few changes to use 10.04, and pull in updated drivers for my intel chipset.
System Hardware
First Thing’s First, this is what my system hardware looks like. Basically a low-end machine from around 2004/2005
lspci00:00.0 Host bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE DRAM Controller/Host-Hub Interface (rev 01) 00:02.0 VGA compatible controller: Intel Corporation 82845G/GL[Brookdale-G]/GE Chipset Integrated Graphics Device (rev 01) 00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01) 00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01) 00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01) 00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 01) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 81) 00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) LPC Interface Bridge (rev 01) 00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE Controller (rev 01) 00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus Controller (rev 01) 00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01) 01:00.0 Mass storage controller: Promise Technology, Inc. PDC40775 (SATA 300 TX2plus) (rev 02) 01:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (LOM) Ethernet Controller (rev 81)
According to the guide, I was going to start with an Ubuntu Server distribution then install X, followed by XBMC. I wanted to save time on the download, so i opted for the Ubuntu Minimal Install CD and downloaded the 32 bit Ubuntu 10.04 “Lucid Lynx” Minimal CD. I then created a bootable USB Key using unetbootin.
Installing Ubuntu Base System
Booting from USB is a challenge for my system – for some reason it just wont do it. Fortunately, since I still have GRUB installed from my broken version of Ubuntu, I can tell grub to boot off of usb:
- Plug the USB key into the computer after loading Ubuntu Minimal on it using unetbootin, and reboot the system
- Press Escape when Grub first loads to enter into the grub boot loader
- Press “c” to enter the Grub Command Line
- Now its a matter of determining what root the USB key is. The easiest way to do this is to type
root
then hit tab. This will list the available roots. I had something like fd0 fd1 hd1 hd2 hd3
- Unplug the USB key, and hit tab again. One of the devices should have disappeared. This device is the USB key (Mine was fd0)
- Enter the following, substituting fd0 for your device name.
root (fd0) chainloader +1 boot
- You should now be presented with the Ubuntu Minimal Install Menu
I allowed for many of the default options when installing Ubuntu. I created a username (that wasnt xbmc – that comes later), as the default user for the system. When it came time to select the software I wanted installed, I selected:
- Basic Ubuntu Server
- OpenSSH Server
- Samba File Server
Xorg Setup
Taken partially from here
sudo add-apt-repository ppa:ubuntu-x-swat/x-updates sudo apt-get update sudo apt-get install xinit x11-xserver-utils
Enable modesetting for the i915 driver (You will need to reboot for this to take effect)
echo "options i915 modeset=1" >> /etc/modprobe.d/i915-kms.conf
Edit the Xorg configuration file at /etc/X11/xorg.conf.
Note that your BusID may differ. Comment it out to have it automatically detected.
Section "Device" Identifier "Configured Video Device" Driver "intel" Option "AccelMethod" "uxa" Option "EXAOptimizeMigration" "true" Option "MigrationHeuristic" "greedy" Option "Tiling" "false" BusID "PCI:00:02:00" EndSection Section "Monitor" Identifier "Configured Monitor" EndSection Section "Screen" Identifier "Default Screen" Monitor "Configured Monitor" Device "Configured Video Device" EndSection
XBMC Setup
I will be using the stable XMBC PPA’s on my system. Don’t forget to install the add-apt-repository tool by installing pkg-config.
For me, the multiverse repos were enabled by default (XMBC needs libfaad0 from them). If this isn’t the case for you, modify /etc/apt/sources.lst to enable multiverse.
sudo apt-get install python-software-properties pkg-config sudo add-apt-repository ppa:team-xbmc sudo apt-get update sudo apt-get install xbmc
Add the XBMC User, and add it to the groups
sudo adduser xbmc --gecos XBMC sudo usermod --group audio,video,fuse,cdrom,plugdev xbmc
Install XBMC-live for automated startup
sudo apt-get install xbmc-live
Logout, then login using the XBMC account, and test start xbmc by running:
xinit xbmc-standalone
Install Alsa
Install Alsa to get sound, run alsamixer to change the volumes as they are all initially zero. Then save the values from alsamixer.
sudo apt-get install linux-sound-base alsa-base alsa-utils sudo alsamixer #unmute PCM, Master and increase the volume sudo alsactl store 0
Mount Additional Drives
I have several drives on my system used for storing video formatted using jfs. By adding these entries to /etc/fstab they will be automatically mounted
/dev/sdb1 /mnt/s320a jfs auto,noatime 0 2 /dev/sdc1 /mnt/s320b jfs auto,noatime 0 2 /dev/sdd1 /mnt/s500 jfs auto,noatime 0 2
auto means that the drive will be automatically mounted, noatime turns off file access time, and the “2″ means that the file will be scanned by fsck on startup. This last option is useful for jfs as they get cranky if they aren’t unmounted properly.
Mount NFS Share
I have one nfs drive shared over the network that I would like to mount:
sudo apt-get install nfs-common
Add another entry to fstab to mount the /mnt/s200 share on the nfs server to /mnt/s200 on the xbmc box:
nfs-server:/mnt/s200 /mnt/s200 nfs _netdev,auto 0 0
Grant XBMC User Power Management Permissions
sudo apt-get install acpid #install acpid to allow shutdown from power button sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.suspend sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.hibernate sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.reboot sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.shutdown sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.reboot-multiple-sessions sudo polkit-auth --user xbmc --grant org.freedesktop.hal.power-management.shutdown-multiple-sessions
Setup Samba
I use samba to easily transfer files from my windows machine onto my media center.
Samba should be already installed as we selected it during the Ubuntu installation, so all we have to do is setup the configuration file.
vim /etc/samba/smb.conf
Modify the security setting in smb.conf to set per-share security (which we will leave wide open).
security = share
I’m not too worried about security, so I’ve made some very liberal shares:
[s320a] comment = XBMC Files path = /mnt/s320a read only = no writeable = yes hosts allow = create mask = 0777 guest ok = yes [s320b] comment = Xbmc Files path = /mnt/s320b writeable = yes read only = no create mask = 0777 guest ok = yes [s500] comment = Xbmc Files path = /mnt/s500 writeable = yes read only = no create mask = 0777 guest ok = yes [yellowface] path = /mnt/yellowface.remote writeable = yes read only = no create mask = 0777 guest ok = yes
Add samba to system startup
update-rc.d smbd defaults 20 21
Additional XBMC Customizations:
ATI Remote Wonder Setup
Disable Windowed Mode
Wifi RADIUS authentication with LDAP on ClearOS 5.2
This guide will help you setup WPA Enterprise authentication using the RADIUS functionality built into ClearOS 5.2.
The tutorial on the ClearOS wiki page is a good starting point to get radius authenticating off of the LDAP user directory, however it stops short of setting up RADIUS encryption which is required when using WIFI.
Preconditions
Please ensure that you have ClearOS 5.2 installed and have completed the guide at http://www.clearfoundation.com/docs/howtos/setting_up_radius_to_use_ldap.
Throughout this guide, it is assumed that ClearOS can be accessed at http://localhost:81. If you are connecting from a remote machine, please updated your url accordingly.
Generating Certificates
Navigate to https://localhost:81/admin/certificates.php.
When initially setting up ClearOS a Certificate Authority should have been created by default. If this isn’t the case, checkout the ClearOS docs for more information.
We are going to be generating a Secure Server Certificate for use with the RADIUS server.
The default Certificate parameters should work, just be sure to include a proper email address in the email field.
After clicking generate, a new certificate should be visible, click view to review the contents of the certificate. Be sure to note the filename as we will be needing that in a moment. Mine is “/etc/ssl/usr-1-cert.pem”
Now that we have the certificates generated, we are going to softlink them into the radius certs folder and update the permissions so the daemon can read them.
cd /etc/raddb/certs #softlink the generated certificate ln -s /etc/ssl/usr-1-cert.pem usr-1-cert.pem #softlink the generated private key ln -s /etc/ssl/private/usr-1-key.pem usr-1-key.pem #update the file ownership chown nobody:radiusd usr-1-cert.pem usr-1-key.pem
RADIUS Configuration
Now that we have the certificates generated, its time to modify the RADIUS configuration files. Remember, the files should have already been modifed as per the wiki article.
/etc/raddb/eap.conf
In the eap.conf file we will be wanting to enable TLS (using our generated certs) and PEAP.
So un-comment out tls and fill in the corresponding information.
The private key file should be set to the key-file softlinked to the /etc/ssl/private directory. (No password is required)
private_key_file = ${raddbdir}/certs/usr-1-key.pem
The certificate file is the cert file softlinked to the /set/ssl/ directory
certificate_file = ${raddbdir}/certs/usr-1-cert.pem
The Trusted root CA list should be the CA certificate for our server
CA_file = /etc/ssl/ca-cert.pem
Additinally, be sure to un-comment the dh_file and the random_file.
tls { # private_key_password = whatever # private_key_file = ${raddbdir}/certs/cert-srv.pem private_key_file = ${raddbdir}/certs/usr-1-key.pem # If Private key & Certificate are located in # the same file, then private_key_file & # certificate_file must contain the same file # name. # certificate_file = ${raddbdir}/certs/cert-srv.pem certificate_file = ${raddbdir}/certs/usr-1-cert.pem # Trusted Root CA list # CA_file = ${raddbdir}/certs/demoCA/cacert.pem CA_file = /etc/ssl/ca-cert.pem dh_file = ${raddbdir}/certs/dh random_file = ${raddbdir}/certs/random # # This can never exceed the size of a RADIUS # packet (4096 bytes), and is preferably half # that, to accomodate other attributes in # RADIUS packet. On most APs the MAX packet # length is configured between 1500 - 1600 # In these cases, fragment size should be # 1024 or less. # # fragment_size = 1024 # include_length is a flag which is # by default set to yes If set to # yes, Total Length of the message is # included in EVERY packet we send. # If set to no, Total Length of the # message is included ONLY in the # First packet of a fragment series. # # include_length = yes # Check the Certificate Revocation List # # 1) Copy CA certificates and CRLs to same directory. # 2) Execute 'c_rehash <CA certs&CRLs Directory>'. # 'c_rehash' is OpenSSL's command. # 3) Add 'CA_path=<CA certs&CRLs directory>' # to radiusd.conf's tls section. # 4) uncomment the line below. # 5) Restart radiusd # check_crl = yes # # If check_cert_issuer is set, the value will # be checked against the DN of the issuer in # the client certificate. If the values do not # match, the cerficate verification will fail, # rejecting the user. # # check_cert_issuer = "/C=GB/ST=Berkshire/L=Newbury/O=My Company Ltd" # # If check_cert_cn is set, the value will # be xlat'ed and checked against the CN # in the client certificate. If the values # do not match, the certificate verification # will fail rejecting the user. # # This check is done only if the previous # "check_cert_issuer" is not set, or if # the check succeeds. # # check_cert_cn = %{User-Name} # # Set this option to specify the allowed # TLS cipher suites. The format is listed # in "man 1 ciphers". # cipher_list = "DEFAULT" }
Setting up peap is easy: just uncomment the directives
peap { # The tunneled EAP session needs a default # EAP type which is separate from the one for # the non-tunneled EAP module. Inside of the # PEAP tunnel, we recommend using MS-CHAPv2, # as that is the default type supported by # Windows clients. default_eap_type = mschapv2 # the PEAP module also has these configuration # items, which are the same as for TTLS. copy_request_to_tunnel = no use_tunneled_reply = no # When the tunneled session is proxied, the # home server may not understand EAP-MSCHAP-V2. # Set this entry to "no" to proxy the tunneled # EAP-MSCHAP-V2 as normal MSCHAPv2. proxy_tunneled_request_as_eap = yes }
/etc/raddb/ldap.attrmap
This file needs an additional line added. Directly before the checkItem $GENERIC$ … line, add
checkItem User-Password userPassword
so the file now looks like:
checkItem User-Password userPassword checkItem $GENERIC$ radiusCheckItem replyItem $GENERIC$ radiusReplyItem checkItem Auth-Type radiusAuthType checkItem Simultaneous-Use radiusSimultaneousUse checkItem Called-Station-Id radiusCalledStationId checkItem Calling-Station-Id radiusCallingStationId checkItem LM-Password sambaLMPassword checkItem NT-Password sambaNTPassword checkItem SMB-Account-CTRL-TEXT sambaAcctFlags checkItem Expiration radiusExpiration checkItem NAS-IP-Address radiusNASIpAddress replyItem Service-Type radiusServiceType replyItem Framed-Protocol radiusFramedProtocol replyItem Framed-IP-Address radiusFramedIPAddress replyItem Framed-IP-Netmask radiusFramedIPNetmask replyItem Framed-Route radiusFramedRoute replyItem Framed-Routing radiusFramedRouting replyItem Filter-Id radiusFilterId replyItem Framed-MTU radiusFramedMTU replyItem Framed-Compression radiusFramedCompression replyItem Login-IP-Host radiusLoginIPHost replyItem Login-Service radiusLoginService replyItem Login-TCP-Port radiusLoginTCPPort replyItem Callback-Number radiusCallbackNumber replyItem Callback-Id radiusCallbackId replyItem Framed-IPX-Network radiusFramedIPXNetwork replyItem Class radiusClass replyItem Session-Timeout radiusSessionTimeout replyItem Idle-Timeout radiusIdleTimeout replyItem Termination-Action radiusTerminationAction replyItem Login-LAT-Service radiusLoginLATService replyItem Login-LAT-Node radiusLoginLATNode replyItem Login-LAT-Group radiusLoginLATGroup replyItem Framed-AppleTalk-Link radiusFramedAppleTalkLink replyItem Framed-AppleTalk-Network radiusFramedAppleTalkNetwork replyItem Framed-AppleTalk-Zone radiusFramedAppleTalkZone replyItem Port-Limit radiusPortLimit replyItem Login-LAT-Port radiusLoginLATPort replyItem Reply-Message radiusReplyMessage
/etc/raddb/radiusd.conf
I found that in the default configuration, the Auth-Type LDAP appeared before the eap in the authenticate section. As a result, the server would cast the request as an LDAP auth type, and fail to parse it as an eap, which would cause the encrypted request from the WIFI access point to fail.
To fix this, I simply swapped the order of the two values, so if the server can’t match against any auth type, it will default to ldap, but most importantly, it will try EAP first.
So the authentication part of the file should look like the following:
authenticate { # # PAP authentication, when a back-end database listed # in the 'authorize' section supplies a password. The # password can be clear-text, or encrypted. Auth-Type PAP { pap } # # Most people want CHAP authentication # A back-end database listed in the 'authorize' section # MUST supply a CLEAR TEXT password. Encrypted passwords # won't work. Auth-Type CHAP { chap } # # MSCHAP authentication. Auth-Type MS-CHAP { mschap } # # If you have a Cisco SIP server authenticating against # FreeRADIUS, uncomment the following line, and the 'digest' # line in the 'authorize' section. # digest # # Pluggable Authentication Modules. # pam # # See 'man getpwent' for information on how the 'unix' # module checks the users password. Note that packets # containing CHAP-Password attributes CANNOT be authenticated # against /etc/passwd! See the FAQ for details. # unix # Uncomment it if you want to use ldap for authentication # # Note that this means "check plain-text password against # the ldap database", which means that EAP won't work, # as it does not supply a plain-text password. #Auth-Type LDAP { # ldap #} # # Allow EAP authentication. eap Auth-Type LDAP { ldap } }
/etc/raddb/users
Comment out the DEFAULT Auth-Type and fallthrough directive, so we aren’t always trying to default to ldap:
DEFAULT LDAP-Group != "radius_users", Auth-Type := Reject #DEFAULT Auth-Type := LDAP # Fall-Through = 1
In Conclusion
You should run service radiusd stop && radiusd -X -A to do your testing with the debug log, as suggested on the ClearOS Wiki.
I found that the radtest still worked as well as authentication from wireless clients using PEAP with MSCHAPv2.
You may want to distribute the usr-1-cert.pem that was generated in the certificates step to wireless clients, however, since we are using password authentication this isnt strictly necessary.
Please let me know in the comments if I have included any redundant or unnecessary steps.
Zend Framework Badword filtering with Cdyne
One of my recent PHP projects had the requirement to filter out inappropriate language from user submitted content. After thinking about the problem briefly, I decided that I didn’t want to be writing the filter myself but, rather, find a third party service that could filter my text for me. By doing this, I eliminated the need to create and maintain a bad-word list, as well as saved the CPU cycles required to actually preform the search-and-replacement (Although, arguably, remote API calls are more expensive anyways).
After some searching I stumbled across the free Cdyne Profanity Filter Service. Not only does this service filter out the standard inappropriate language that you would expect, it also doesn’t produce false negatives (eg: the hello isn’t filtered for containing the word hell), and it has fairly robust phonetic character matching to catch things like a$$. The Cdyne service is exposed as a Soap WSDL so easy interfaces to languages other than PHP are possible.
I ended up writing a Zend Framework based Soap Client Service for the Cdyne filter, and I figured I would share it with any others who are looking to do filtering. In the following zip, there is the Service class, along with some unit tests demonstrating the use of the class methods. You should be able to rename the Zext_Service_Cdyne_ProfanityFilter class to one of your choosing if you do not like the pseudo namespacing I’ve used. Check out Cdyne’s wiki for more info.
Enable Xdebug HTML Output
I recently updated to Ubuntu 10.04 which includes PHP-5.3 by default. Ever since that upgrade I’ve had trouble with Xdebug not displaying HTML in its output: instead it would output human-readable text which becomes quite unreadable when rendered in the browser without <pre> tags.
Turns out when PHP was upgraded, the php.ini file was replaced, and in the new version, it has the production value for html_errors which is Off. Once I set html_errors=On in /etc/php5/apache, colourful HTML-formatted output was producted by Xdebug.
For the Curious reader, I’m using the following xdebug configuration file, which I’ve located in /etc/php5/conf.d/xdebug.ini to prevent any php.ini changes from overwriting my Xdebug directives. Please note that this is a slightly modified version of the default xdebug configuration included in wamp, which I find to be quite nice:
zend_extension="/usr/lib/php5/20090626/xdebug.so" xdebug.remote_enable=1 xdebug.remote_host="127.0.0.1" xdebug.remote_port=9000 xdebug.remote_handler="dbgp" xdebug.remote_mode=req xdebug.profiler_enable=1 xdebug.profiler_output_dir="/tmp/xprofile/" xdebug.collect_params=On xdebug.show_local_vars=On