I put a bunch of Xcode text macros I use for writing Objective-C on GitHub, together with a cheat sheet generator:

http://github.com/liyanage/xcode-text-macros

You’re welcome to contribute additional macros to the collection.

Apple Generic Versioning (AGV) is a system that helps with software project version number management in Xcode projects. Adopting it has some benefits such as stamping the version information into the binaries in a way compatible with the what tool.

I thought about using it for my personal projects in the past, but was never comfortable with the agvtool utility manipulating files such as the Xcode project file directly. I thought that I would corrupt these files when I forgot to close the project in Xcode before bumping a version.

I’m now much more comfortable with that thought because I realized how well Xcode handles external changes to its open files. I realized that when I moved my projects to Git and started using local branches. At first I avoided switching between those branches while the project was open in Xcode for the same reason, but when I actually tested what happens when I switch back and forth I saw that Xcode handles it well:

Obviously you shouldn’t do this with files that have unsaved changes. Xcode seems to store all changes to the project settings to its project file right away, so you usually only have to check for unsaved source code files.

Now that I’m confident that I can’t get myself into trouble with agvtool (at least not this particular kind of trouble) here’s how I set it up:

First I enable AGV on the project level. There are two settings, setting the versioning system to AGV and setting the initial version number:

Make sure you set this on the project level and in “All Configurations.” There are some other settings on the target level, but Xcode should set those to sensible default values when you flip the switch to enable AGV.

Next I replace all occurrences of the project version number in the Info.plist file with a placeholder:

<key>CFBundleGetInfoString</key>
<string>Checksum version ${CURRENT_PROJECT_VERSION}, Copyright ...</string>
<key>CFBundleShortVersionString</key>
<string>Checksum version ${CURRENT_PROJECT_VERSION}</string>
<key>CFBundleVersion</key>
<string>${CURRENT_PROJECT_VERSION}</string>

From now on I can bump versions on the command line like this:

agvtool new-version 2.9

Xcode will display its “changed externally” message when it’s coming to the front again and after it reloads the project you can build with the new version number.

These two articles cover Xcode project versioning in much more detail, including alternatives to AGV:

http://www.dribin.org/dave/blog/archives/2006/08/02/versioning_os_x_apps/
http://chanson.livejournal.com/125568.html

Ted Stresen-Reuter, author of a big, popular BBEdit clipping set for PHP, Carlton Gibson, and I recently combined all our BBEdit clipping sets into one big collection, and the result is now on GitHub:

http://github.com/tedmasterweb/bbeditclippings/

This was a good opportunity to clean up the sets, especially the one for Perl. It got lots of new clippings and a more modern coding style in existing clippings.

I also realized that the sets on my site were always hopelessly out of date because I never published new versions when I improved and added clippings. With the code now available on GitHub, publishing is much easier for me and users of the sets can always update to the most recent version with a quick git pull.

You’re welcome to contribute additional clipping sets for other programming languages to this collection. If want to add your set, use GitHub’s fork feature to clone our project, then add your stuff and let us know about it.

Here’s an updated version of checklibs.pl, a developer tool that lists dynamic library dependencies of Mac OS X binaries. I introduced it in an earlier article.

The new version colorizes all libraries that are outside of the /System and /usr/lib directories and therefore not part of a standard Mac OS X installation.

The color helps spotting non-standard libraries. For each one, you have to determine if the dependency is a build mistake (hello MacPorts or Fink in $PATH) or if it’s OK because the library will be shipped and installed along with the binary.

The script is now under revision control as part of my Mac OS X shell script collection on GitHub:

http://github.com/liyanage/macosx-shell-scripts

I got a notice from Amazon telling me to change my Album Artwork Assistant application within 60 days so that it signs its search requests to the Amazon AWS product information web service. I use this service to find album artwork images.

Amazon provides some notes and examples for Java, Perl and C#, but not for Objective-C. I wrapped my signed request URL generation in a class that other Cocoa developers can reuse. It’s available here:

SignedAwsSearchRequest.h and SignedAwsSearchRequest.m

Example usage can be seen in the unit test file TestSignedAwsSearchRequest.m

I have written about clearing Apple’s Internet download quarantine flag from files to get rid of the security warning before.

Recently Apple pushed Xcode documentation updates frequently, and I somtimes had to clear the flag from other HTML files all over the harddisk, not just in /Developer. The old way was just too slow for frequently changing and/or big sets of files:

find /Developer -type f -name '*.html' \
-exec sh -c 'xattr "$0" | grep -q quarantine && xattr -d com.apple.quarantine "$0"' {} \; -print

This spawns lots of subprocesses and it turns out that xattr is not a native binary but a Python script, which means that the Python interpreter is fired up twice for each file.

It takes ages this way so I rewrote it to loop in Python and use Python’s native xattr module:

#!/bin/bash
#
# Remove the quarantine extended attribute from all
# developer documentation HTML files to get rid
# of the "downloaded from the Internet" warning
#
# Marc Liyanage / www.entropy.ch
#

[ $UID -eq 0 ] || { echo $0 must be run as root; exit 1; }

find /Developer -type f -name '*.html' | python <(cat - <<EOF
#!/usr/bin/env python

from xattr import *
import sys
import string

attr = 'com.apple.quarantine'

for file in sys.stdin:
	file = string.rstrip(file, "\n")
	if (attr in listxattr(file)):
		removexattr(file, attr)
EOF
)

This is a lot faster.

Here’s my take on displaying Git and SVN status information in the Bash prompt:

# Prompt setup, with SCM status
parse_git_branch() {
	local DIRTY STATUS
	STATUS=$(git status 2>/dev/null)
	[ $? -eq 128 ] && return
	[[ "$STATUS" == *'working directory clean'* ]] || DIRTY=' *'
	echo "($(git branch 2>/dev/null | sed -e '/^[^*]/d' -e 's/* //')$DIRTY)"
}

parse_svn_revision() {
	local DIRTY REV=$(svn info 2>/dev/null | grep Revision | sed -e 's/Revision: //')
	[ "$REV" ] || return
	[ "$(svn st)" ] && DIRTY=' *'
	echo "(r$REV$DIRTY)"
}

PS1='\u@\h:\W$(parse_git_branch)$(parse_svn_revision) \$ '

Put this into your ~/.bashrc script and you should see the Git/SVN status in your prompt if your working directory is a sandbox.

For SVN, it displays the current revision:

liyanage@primavera:foobar(r9851 *) $

For Git, it displays the current branch:

liyanage@primavera:build-entropy-php(php-53 *) $

For both, a * means that there are local changes to the working directory

An alternative version of parse_svn_revision(), based on David’s comment below:

parse_svn_revision() {
	local REV=$(svnversion 2>/dev/null)
	[ $? -eq 0 ] || return
	[ "$REV" == 'exported' ] && return
	echo "($REV)"
}

With that, the prompt looks like this:

liyanage@primavera:foobar(10027:10028M) $

This version is faster on large SVN trees. The original one above sometimes introduces a noticeable delay before the prompt appears.

If you’re a Perl developer on Mac OS X Leopard, then you’ve probably heard about this issue caused by Security Update 2009-001:

...the latest Mac OS X Security Update 2009-001 apparently breaks your Perl, if you

• are using Mac OS X 10.5 (Leopard)
• and are using Perl that comes with Mac OS X
• and have upgraded several core modules with CPAN in the past

I read about it a while ago but when I tried it on my machine at the time, I didn’t see the symptoms described in the article, even though I use CPAN a lot, so I thought that my system wasn’t affected.

Today, however, I had to track down a strange, hard to find bug where I would get messages like this:

Modification of a read-only value attempted

At first I thought I made some Perl beginner mistake, but in the end it turned out that I did indeed have a broken List::Util module with a version mismatch between the XS binary and the Perl module code. After downloading, compiling and installing a fresh copy of Scalar-List-Utils from CPAN, everything was back to normal.

Putting it here in the hope that it saves some other googling Perl developer some time :-)

I converted my personal Subversion repository to Git and collected a few notes.

My repository contains the source code of my Mac OS X projects, but also a lot of one-off test programs and experiments. It is a very basic repository; there are no branches at all (maybe I'll use them more with Git) and very few of my projects used tags, so the conversion was straightforward. Because there was so little information to preserve, this is not representative of an SVN -> Git conversion in general, but the notes might still be useful for other people converting their personal repos.

I had about 50 projects in Subversion, some big and some with just one or two files. Each project was in its own subdirectory below the repository root. They were unrelated, so I wanted to separate the projects into their own Git repositories during the conversion.

Converting Projects With Tags

I converted each of the few projects that did use tags manually, like this:

SVN_BASE=svn+ssh://host/path/to/svnroot
PROJECT=album-artwork-assistant
git svn clone --trunk="$SVN_BASE/trunk/$PROJECT" --tags="$SVN_BASE/tags/$PROJECT/versions" "$SVN_BASE/trunk/$PROJECT"
cd $PROJECT

To convert the SVN tags to real Git tags I ran a modified version of a script by Frank S. Thomas:

#!/bin/bash
GIT=/opt/local/bin/git
for branch in $($GIT branch -r); do
    if [ $(echo $branch | egrep "tags/.+$") ]; then
        version=v$(basename $branch)
        subject=$($GIT log -1 --pretty=format:"%s" $branch)
        export GIT_COMMITTER_DATE=$($GIT log -1 --pretty=format:"%ci" $branch)

        echo "Tag $version [Y/n]?"
        read yesno
        if [ -z $yesno ] || [ $yesno = "Y" ]; then
            $GIT tag -a -f -m "$subject" "$version" "$branch^"
            $GIT branch -d -r $branch
        fi
    fi
done

This next step converts svn:ignore properties to .gitignore files

git svn create-ignore
git commit -m 'added .gitignore converted from svn:ignore'
cd ..

All of this gave me a working copy repository in $PROJECT. I converted it to a bare repository like this:

git clone --bare $PROJECT

The bare repo was now in $PROJECT.git

Converting Simple Projects Without Tags

For all the other simple projects without any tags I used this script:

#!/bin/bash

WD="$PWD"
GIT="/opt/local/bin/git"
SVN_TRUNK=svn+ssh://host/path/to/svnroot/trunk
PROJECTS=$(svn list "$SVN_TRUNK" | sed -e 's#/$##')
[ -d logs ] || mkdir logs

for project in $PROJECTS; do
	cd "$WD"
	[ -d "$project.git" ] && continue

	LOG="$WD/logs/$project.log"
	echo "\nconverting $project\n"

	$GIT svn clone "$SVN_TRUNK/$project" > "$LOG"
	cd "$project"
	$GIT svn create-ignore >> "$LOG"
	$GIT commit -m 'added .gitignore converted from svn:ignore' >> "$LOG"
	cd ..
	$GIT clone --bare "$project" "$project.git" >> "$LOG"
	rm -rf "./$project"
done

Move to Server and Checkout Local Repositories

Together with the manually converted ones, this resulted in about 50 bare Git repositories, one per project. I moved them all to a server into a common project root directory and then cloned them back to my computer as local working copies:

for project in $(ssh myserver ls /path/to/git/projectroot); do
    git clone -v ssh://myserver/path/to/git/projectroot/$i
done

Setting up gitweb

I got Git for Mac OS X via MacPorts, both on my client and on the server. It comes with the gitweb web viewer so I activated that on my server with the following Apache configuration:

Alias /git /opt/local/share/git-core/gitweb
<Location /git>
    DirectoryIndex gitweb.cgi
    AddHandler cgi-script cgi
    Options +ExecCGI
</Location>

I had to configure the project root directory in gitweb.cgi:

our $projectroot = "/path/to/git/projectroot";

Now it’s visible externally at http://www.entropy.ch/git/

Rewriting History

After the conversion, the old commits generated by git svn had auto-generated committer e-mail address and name information which differed from my Git configuration values. To clean up and unify this information in the old commits I ran these commands in each working copy:

git filter-branch --env-filter \
    "export GIT_AUTHOR_NAME='Marc Liyanage' \
    GIT_AUTHOR_EMAIL='email@example.com' \
    GIT_COMMITTER_NAME='Marc Liyanage' \
    GIT_COMMITTER_EMAIL='email@example.com'"
git push --force

github

One nice benefit of using Git is that I get to host code on github for free. I push some of my more recent projects there from time to time.

MySQL Workbench is coming to the Mac (and Linux). This program is a great database design tool, I use its Windows version to model relational databases visually. It gives me two kinds of very useful results:

• DDL SQL files that I can send directly to the MySQL database server to initialize my database schema
• database model diagram vector images that I can use for high quality output technical documentation and also for training and discussion purposes

Here’s MySQL’s announcement of the Mac version:

http://dev.mysql.com/workbench/?p=192

As they note, it’s a very early preview that’s probably not yet ready for production, but it already looks great and opens and displays my model file from the Windows version fine.

When I say it looks great I’m really a bit surprised because I expected a really ugly port like so often with cross-plattform applications (I’m thinking of the ugliness that is often QT and I’m looking in the general direction of Google Earth when I say this).

I saw some visual glitches in other parts of the interface, for sure, but overall it’s a positive first impression.

At work I recently gave a presentation about the practical aspects of public key cryptography with the OpenSSL toolkit. This is an expanded transcript of the presentation, published here on my blog in the hope that it’s useful for a broader audience.

There are four parts covering these major topics:

• Part 1: Public Key Cryptography
• Part 2: Public Key Infrastructure (PKI)
• Part 3: Inspecting and Creating X.509 Certificates
• Part 4: OpenSSL Bag of Tricks

You are reading part 2, about Public Key Infrastructure (PKI). This part shows the practical problems with simple public key cryptography, and how a public key infrastructure attempts to solve them. While part 1 was fairly technical, this part is about real-world organizational structures and trust relationships. Part 3 will again be more technical, showing how these real-world concepts are mapped to systems and software.

Some things in this part are based on the German book “Kryptografie und Public-Key Infrastrukturen im Internet”. A revised English edition is available as “Cryptography and Public Key Infrastructure on the Internet”. Both are probably a bit dated by now, so you should look for newer books.

Practical Problems With Plain Public Key Cryptography

There are a few practical issues with the basic operations provided to us by plain public key crypto. The next few sections highlight some of them:

• Key authenticity
• Key revocation
• Key validity periods

There are other issues not discussed here, such as non-repudiation and policy enforcement.

For public cryptography to be useful in real-world use on a large scale, these issues need to be addressed, and that is what a Public Key Infrastructure, or PKI, is about.

Key authenticity

Part 1 didn’t go into the details of how two parties who want to exchange encrypted or signed messages exchange their public keys, but these details are critical.

One possibility is to exchange the keys at a face-to-face meeting. This approach is not feasible on a large scale, and public key cryptography with its asymmetric nature is interesting exactly in a large scale scenario.

In the absence of a secure communications channel for key exchange between the two parties, another possibility is to exchange the keys over an open, untrusted network like the Internet. The problem with this is that the recipient of the key can never be sure if the key is authentic.

A third party C, with the ability to intercept messages on the untrusted network, could intercept the public key in transit from the sender A to the recipient B and note it. C could generate a new key pair and send the public key to B, pretending to be A. If B then uses this public key to send an encrypted message to A, the third party could again intercept the message, decrypt it with the private key it generated to read the message, and then re-encrypt the message using A’s real public key and send it to A, pretending to be the legitimate sender B. It would work the same in the other direction and it’s called the man-in-the-middle attack (MITM).

A public key infrastructure should make it possible to verify a public key’s authenticity.

Key revocation

If a private key is compromised, we want our communication partners to stop using the related public key, otherwise confidential information is sent around and people other than the intended recipient are able to decrypt it.

A public key infrastructure should make it possible to revoke previously distributed public keys.

Key validity periods

Encryption keys should be replaced in regular intervals. One reason is that the longer a key is in use, the more material is available for cryptanalysis. Another reason is to limit the damage caused by a private key being compromised without detection.

A public key infrastructure should provide means to specify a validity period for a public key.

Trust Relationships

Ultimately, these issues are about trust and trust relationships between the involved parties. There are different types of trust relationships, and this section shows how well each one addresses the problems.

Direct Trust

This model could also be called “no infrastructure”. It’s the basic face-to-face meeting case from above, where the parties exchange their public keys directly.

It’s difficult to do with parties who are total strangers, and between difficult and impossible to do with lots of communication partners.

Key revocation means directly contacting the other party and telling them to stop using the old key.

There is no support for formal validity periods.

Web of Trust

This model is an extension of direct trust. If participants A and B have previously exchanged their keys, both can now act as introducers and sign (and forward to the other party) the public keys of all other partners they know directly. Each can add the public keys obtained this way to its list of known, trusted keys.

So if A knows B, and B knows C, B is the introducer and can:

• sign C’s public key and send it to A
• sign A’s public key and send it to C

This enables A and C each to verify the new public keys using B’s public key, which they already know and trust, and begin exchanging secure messages directly, knowing that their public keys for each other are authentic.

Now that A and C trust each other, they can again sign keys of partners they know directly, and so on, thus creating and extending the web of trust.

It is usually not just the public key that is signed, but a whole package of information that includes the public key, the subject’s name, e-mail address etc. This whole information package, when signed, is also called a certificate. Make a mental note of this, because it’s important: A certificate contains a public key and additional information, and somebody signed the whole package with a private key that’s unrelated to the public key in the package.

The party who signs the bunch of data, and therefore issues the certificate, vouches for the entire information, and not just for the public key. This means that the signing party needs to verify all information that it signs. For things like e-mail addresses, that can be through an automated process, but for things like a company or personal name, it usually requires checking official records such as commercial registry papers, passports, ID cards etc.

Handling revocation, expiration and scaling to large users bases can be difficult, as can be getting introduced into a web of trust. Wikipedia’s article on web of trust has more information.

Web of trust is the trust model used by PGP.

Hierarchical Trust

In this model, all parties agree on one or more trusted, independent authorities to issue certificates. These authorities are called certificate authorities (CAs) or trust centers.

A CA’s own certificate, which the parties need in order to verify the certificates issued by that particular CA, is called a root certificate and it is distributed widely, e.g. as part of client programs such as e-mail clients or web browsers or directly in operating systems.

Here are parts of the root certificate lists of Mac OS X and Windows XP:

Because literally millions of users trust these CA certificates, the non-disclosure of the associated private keys is absolutely critical. The CA’s organizational structures and operational processes are designed to prevent a compromise of these very important private keys. A CA has to document the quality of these structures and processes by passing an audit before its root certificate is accepted by operating system and other client software vendors. One requirement is the use of Hardware Security Modules conforming to the FIPS 140-2 standard for the generation, storage and use of the CA’s sensitive key material.

Once a party trusts a CA (by having its certificate with its public key in the list of trusted root certificates), it can immediately validate the probably very large number of certificates issued by that CA now and in the future. A CA can also sign intermediate signing keys of other CAs, and they can in turn do the same, which creates the hierarchical trust structure that gives this model its name.

With a CA in place, if two parties both trust that CA and want to communicate securely with each other using public keys certified by that CA, they both have to get a certificate. They generate a key pair and prepare a certificate request. The request is the “bunch of data” mentioned above and it consists of the public key along with all other information required by the CA, according to the CA’s policies.

These policies can vary according to the level of certification, and the price of the certificate usually depends on that level. It could be that only an e-mail address is certified and the certification is free. In other cases, commercial banking-grade certificates are issued that certify not only the name of a web site but also the organization name, after extensive review of legal proof of incorporation and domain name ownership.

After following its procedures to verify that all information presented in the certificate request is valid, the CA signs the public key and other information in the request with its private key. The result is the certificate, which is sent back to the party that requested it.

Once both parties have their certificates, they can exchange them, and they can do this even over insecure public networks because both will be able to verify the certificate’s authenticity using the CA’s root certificate, which they both already know and trust.

So how well does this trust model deal with our practical issues?

Regarding key revocation: With the centralization resulting from a limited number of trusted certificate issuers (my Mac OS X system keychain lists about 150 trusted root CAs), certificate revocation in the form of certificate revocation lists (CRLs) becomes feasible. A CRL is a list of serial numbers of certificates that the CA has revoked. When a CA issues a certificate, it includes the location of this CRL. When a client needs to verify a certificate, it can check if it is on the list of revoked certificates.

For this setup to be useful, the list needs to be updated frequently. On Mac OS X, there’s a daemon called ocspd that does just that. Its name already suggests that it also provides support for an alternative to CRLs, the Online Certificate Status Protocol (OCSP). With OCSP, the client doesn’t download a potentially large list from the CA’s servers from time to time. Instead, it asks specifically about the validity of one particular certificate that it needs to check in real-time. This results in up-to-date validity information and reduces network and storage requirements, a benefit especially on mobile devices.

Other things like validity periods, mandatory attributes etc. are not only possible but actively enforced by the issuing CA according to its policies.

Conclusion

Of these models, only the hierarchical trust model is considered a Public Key Infrastructure (PKI), so that’s what people refer to when they talk about PKI.

The technical standard in use for this model on the Internet today is X.509, and more specifically the IETF’s variant PKIX.

The next part in this series, part 3, gets back to the practical technical level and shows you how you can inspect and create X.509 certificates with the OpenSSL toolkit.

Google Custom Search Engine allows you to set up search engines with a narrow focus on specialized topics. It features powerful search options and match patterns that let you specify precisely what should and should not end up in the search results.

I set up a custom search engine covering web sites, blogs and mailing list archives related to Mac OS X software development and gave it the following short, easy to remember link:

http://bit.ly/macdev

Try it out right here with something Mac OS X related, for example “IKImageBrowserView”:

The search engine has “Refinements” to focus on results of a particular type, such as reference documentation, mailing list postings etc.:

Usage

Bookmarklet

For frequent use, here is a bookmarklet:

Mac Dev Search

Drag it to your browser’s bookmarks bar:

Then select some text on a web page and click the bookmarklet button you created to search for the text in the Mac development search engine.

LaunchBar/Quicksilver etc.

If you use an application launcher such as LaunchBar or QuickSilver, you can define a search template. In LaunchBar I use this search template with a name/abbreviation of “macdev”:

http://www.google.com/cse?cx=003121735520996018220%3A_bu0g4hqtke&ie=UTF-8&sa=Search&q=*

Embed

You can embed the search box into your own pages. Here’s Google’s code do do that:

<form action="http://www.google.com/cse" id="cse-search-box">
  <div>
    <input type="hidden" name="cx" value="003121735520996018220:_bu0g4hqtke" />
    <input type="hidden" name="ie" value="UTF-8" />
    <input type="text" name="q" size="31" />
    <input type="submit" name="sa" value="Search" />
  </div>
</form>
<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=cse-search-box&lang=en">

Coverage

The configuration of the sites it covers currently looks like this:

Let me know if you know additional useful sites that the custom search should cover.

At work I recently gave a presentation about the practical aspects of public key cryptography with the OpenSSL toolkit. I showed many openssl command line examples and when I was asked for a transcript I decided to expand the material and publish it here in the hope that it’s useful for others too.

If you see broken examples or broken English, please let me know.

Thanks to Kaspar Brand for reviewing the material and providing additional information and links.

There are four parts covering these major topics:

• Part 1: Public Key Cryptography
• Part 2: Public Key Infrastructure (PKI)
• Part 3: Inspecting and Creating X.509 Certificates
• Part 4: OpenSSL Bag of Tricks

This is part 1, about Public Key Cryptography. This part shows how to create RSA private and public keys and then use those keys to encrypt/decript and sign/verify information with openssl.

This assumes that you know what public key cryptography is about. If you don’t, read the Wikipedia article, especially the postal analogy. Wikipedia summarizes:

Public-key cryptography is a method for secret communication between two parties without requiring an initial exchange of secret keys. It can also be used to create digital signatures. Public key cryptography is a fundamental and widely used technology, and enables secure transmission of information on the Internet.

It is also known as asymmetric cryptography because the key used to encrypt a message differs from the key used to decrypt it. In public key cryptography, a user has a pair of cryptographic keys—a public key and a private key. The private key is kept secret, while the public key may be widely distributed. Messages are encrypted with the recipient's public key and can only be decrypted with the corresponding private key. The keys are related mathematically, but the private key cannot be practically derived from the public key.

Creating Keys

So we need to create a key pair and we start with the private key.

The openssl command line utility provides a lot of functionality. To keep it manageable, this functionality is broken down into various subcommands. The subcommand to create an RSA private key is genrsa. In its simplest form, it works without parameters:

openssl genrsa

Generating RSA private key, 512 bit long modulus
....++++++++++++
.++++++++++++
e is 65537 (0x10001)
-----BEGIN RSA PRIVATE KEY-----
MIIBPAIBAAJBALWv2VcerryVqaj0AZ/tTLUa3L+sf/CVl27gul4l5B/F/FbjpYB1
4HOuewKCXqJqUiNJechliFsi7XqWedOta4MCAwEAAQJBAIaiwic6GHitW2VVNGMD
4WUPe+gZ91EcOXSz2cdswsj7l+LrjiNkOVClQANfZiP/d9YSFsk8WzyHnTtlbgun
VeECIQDvnwNdWJ6Q3y+46s+d2FSe1P22lb5qMuSQYUhDLqZScwIhAMIbFFHW2eKB
3mRCqY6oSY9pK1r6QSssL/QGuIMKe26xAiEA7JgAFXn7zqVgFGBcsMi5/L3m6RH/
mhI4FcrIM/VqqbsCIEgklGrOdfDv395XwHlbJuv8ZLbNqIcJR1FlZhKeRL4hAiEA
nehFFepANrjgjPU6iqM/5T6454mqDpQmOQzZHlDh+Ys=
-----END RSA PRIVATE KEY-----

That outputs the private key in a plain text format called PEM on the console. We’ll want it in a file instead:

openssl genrsa > private-key.pem

Another openssl subcommand called rsa, and specifically its option -text, lets us inspect what exactly is in the private key:

openssl rsa -text < private-key.pem

Private-Key: (512 bit)
modulus:
    00:bc:42:41:6e:cd:07:28:0d:0e:27:dc:79:e4:0e:
    fc:9e:f8:74:5d:db:42:82:c1:f9:97:9a:9a:0b:d2:
    2d:76:e0:04:74:34:5b:de:c6:67:58:47:5d:12:84:
    e6:a1:d8:dd:94:67:71:d7:2a:00:4d:27:67:68:5f:
    ef:a3:94:e8:e9
publicExponent: 65537 (0x10001)
privateExponent:
    77:dd:b7:cb:d4:bf:b9:c0:96:42:cb:1e:d1:16:b9:
    c2:7e:6f:99:53:02:06:3f:7b:ac:0b:c4:09:c2:c9:
    2f:e6:5a:a3:e3:a7:0c:bd:ef:59:00:55:8c:79:14:
    35:82:3e:4f:41:0c:5a:cf:e7:b2:09:9f:ba:3a:7c:
    be:a7:b0:01
[...]

The private key is sensitive information so it shouldn’t lie around on the file system unprotected like that. openssl’s -des3 option encrypts the key with a password:

openssl genrsa -des3 > private-key.pem

Generating RSA private key, 512 bit long modulus
......++++++++++++
....++++++++++++
e is 65537 (0x10001)
Enter pass phrase:
Verifying - Enter pass phrase:

The key now looks like this:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,93CD4E9C3BD610A5

/gwqg7Vp+y9mN/J8qyzOxBUX5lnVMpwYBe/EMPDcC1KDSj//vSfcRzta/lxT8MPS
7JA+wYajxTNnRoGZCip73GgtfHjiHpf7LlNPHTinDgD0La8ZmNET507ciRmOgRCZ
y8+mmaFUcsgBmn5e/mtFJbFnbv/MCwIEpQ0ootfCgwoqicKG6y+D/3qoR1TxTUTE
sV00+n7BN6s7yZftmaXZANVz8DBbhnAgOQZBHJkAQ0vedIlL2wOWm8MdOUOxeGTY
Y+AzaOy9FJg6rGdiuB4xGKUUXzrVw8ophD2q3cK+ha4RxQiWPzrhLZsjcffXMLEF
U3FLG+ivqDd+v1EgpO213kbPaFH0IjzoWsR8vwEjY6N9T8wYha0mEfAM2HDL2kuk
gohlHnrkFdZMbuqc8jQ6cAhcFrRgzAM+8Bm+yc4Duvw=
-----END RSA PRIVATE KEY-----

Note the password prompt and the additional header lines indicating the encryption.

With this protection, openssl asks for the password every time the key is used. For example, the rsa -text command from above now behaves like this:

openssl rsa -text < private-key.pem

Enter pass phrase:
Private-Key: (512 bit)
[...]

Sometimes there’s a reason to store the keys unencrypted, for example when systems like a web server need to be able to load the key automatically. For clarity, I also omitted the encryption in the examples below.

genrsa’s default settings generate private keys with 512 bits but more would be better. To generate longer keys, add the number of bits as argument:

openssl genrsa 1024 > private-key.pem

This key is longer:

-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDLVqzJbJMPxpUnUlhv8x1qd6dM2peHYpP3PbTpxBFYXmfPzqMt
XYpdpcLPHP/DGuabkWcfUtlfBuqFy/03aVI24xchYUAvuiJsfPbLErufFfYwUdW/
Bp0xIp/pptjzyOsx86FCroGQ/8VWBnMkmeG39i8sfXYSQt5Jptr1OnSJoQIDAQAB
AoGAcrO1Cnalyothzemkm5oPqORQSokwWx2fjCOR7yA8DJZMhyvwXoHasJL1/nFE
UWHWXkE6Y7APkIxNNSZp850E05G42IEYuvEDYp74WKL9Cf+db4t/JJeR+3Vcrila
poMhkAevuM4QWPmacTu58B2CiJI5GiQxf7w2m4Qm2EtMpuUCQQDvYDmOYRwcZF4Z
ScBeKCI8Atekr7gmfAo83IxRJ4goLtp12/ou6frIFD/dG9dLkWCtTMcD16Bs8GjI
Mvo8fkP/AkEA2XXBl71VCu+iAEysVwL0fDiyRNRs4rOWvH2oHxmP7cQKetDElX50
6ro9/4yU9bbnBddrDeoQKFS9dVl1O66yXwJBAIn9K1CjNEQ9q2zice0VL57ueIbr
8LjmmjQ5Yv3JTxjy4WY9l7wBj1pVKfd4/CQIuvVLpBHX9Be4gn6dvFiw5NsCQQDA
74tdt6vWUhonrd9SK9sCw8LW8qnQ0YmxnsMlJKbgS4kIwmVIRbQx9h+/tdkL0MVU
F3ZyF/NR9S1LCp3K4Ap9AkEA4Ee88r/hkRih3C2sajIpXire7h7rhry5NaNzND7B
q2FpVqaJfoJvhsTIX8ju/pzF6WbVL6VjrY4OJ3miLWGBEg==
-----END RSA PRIVATE KEY-----

Now we’ll extract the public key into a separate file, again with the rsa subcommand:

openssl rsa -pubout < private-key.pem

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDIxa9c7tYDvCL5ZCk5FGT2Zlo2
nYLEpX1KocTjfggXjrnL6+92HZMJs2jcbxp6Q4lyMPhlGP3t7/DCBpXM39v7w3U7
HRy0GAxa7tWICCnvXZRoI2BKF3bZDuWSeNyHXJtkJwnId6udn7eLj2q5TdTxc8On
JpCKjheJ/xs9fFHmSwIDAQAB
-----END PUBLIC KEY-----

And again we’ll want this in a file:

openssl rsa -pubout < private-key.pem > public-key.pem

A tip regarding openssl’s man pages: Each subcommand has its own manpage, so to get information about genrsa you would type man genrsa and not man openssl.

Using the keys

Now we have a key pair. As noted above, this allows us to:

• encrypt and decrypt a message to make sure that it is not read in transit
• sign and verify a message to make sure that it is not altered in transit

It is possible to combine these operations, ensuring that a message is neither read nor altered in transit.

Encryption and Decryption

To send an encrypted message, the sender encrypts the message with the recipient’s public key (what we stored in the file public-key.pem above). How exactly the sender gets this key is not discussed here but it is important and the topic of the next part in this series, “Part 2: Public Key Infrastructure”. For now let’s assume that sender and recipient met in person and exchanged their public keys.

The recipient uses his or her private key (what we stored in the file private-key.pem above) to decrypt the message.

Our example message content to be protected is the string “Alice in Wonderland”, stored in the file test.txt:

echo "Alice in Wonderland" > test.txt

Let’s encrypt this content. The openssl subcommand is rsautl:

openssl rsautl -encrypt -pubin -inkey public-key.pem < test.txt > test-encrypted.txt

test-encrypted.txt now contains the encrypted message as unreadable binary data. This block of data is what’s sent over an insecure channel to the recipient.

The recipient also uses rsautl to decrypt the message, using the private key:

openssl rsautl -decrypt -inkey private-key.pem < test-encrypted.txt

this outputs

Alice in Wonderland

What if we want to send a much longer message? Lets’s try the list of English words in /usr/share/dict/words:

openssl rsautl -encrypt -pubin -inkey public-key.pem < /usr/share/dict/words > test-encrypted.txt

This fails with a “data too large” message as rsautl can’t encrypt arbitrary amounts of data:

RSA operation error
37951:error:0406D06E:rsa routines:RSA_padding_add_PKCS1_type_2:data too large for key size:rsa_pk1.c:151:

To handle this, we use a clever combination of the public key encryption we already have with a symmetric key algorithm, for example the AES block cipher.

We first encrypt the data with the symmetric key algorithm, which can handle large amounts of data. Such an algorithm requires a key too, and sender and recipient use the same key to encrypt and decrypt the message, which is why it’s called symmetric. Normally sender and recipient need to agree on a shared secret key before exchanging messages, but in this use case, the sender just makes up a random encryption key for the symmetric algorithm and then encrypts the message content with the key. Next, the sender encrypts the made-up, random key with the recipient’s public key. The symmetric algorithm key is small enough (comparable to our “Alice in Wonderland” in size) that rsautl can process it.

The message we send to the recipient consists of the large data block encrypted (symmetrically) with the random key and the (asymmetrically) encrypted random key itself. Both are unreadable to someone intercepting the message in transit.

This combination of symmetric and public key encryption has another advantage: Processing the same amount of data with a public key algorithm is much more computationally intensive than with a symmetric key algorithm. Many encrypted real-world communications protocols such as SSL/TLS and S/MIME use this combination.

Let’s work through an example. First we decide on (make up) a random key for the symmetric encryption and stick that key into a file. We’ll use the incredibly hard to guess “foobar”:

echo -n foobar > test.txt

Then we encrypt the large word list file with the AES cipher using this password and store the encrypted data in another file:

openssl enc -aes-128-cbc -salt -pass file:test.txt < /usr/share/dict/words > words.dat

(Technically, “foobar” is not the key but a password from which openssl derives the actual key used by the algorithm.)

Now we asymmetrically encrypt the small password file with the recipient’s public key. You already know how to do this, because it is exactly what we used above:

openssl rsautl -encrypt -pubin -inkey public-key.pem < test.txt > test-encrypted.txt

It’s exactly the same except that instead of “Alice in Wonderland” we encrypt “foobar”.

We now have the two parts that together make up the protected message, they are in the two files words.dat and test-encrypted.txt. These two have to be sent to the recipient, and if someone intercepts them in transit, they are useless.

For the recipient of the message, the first step is to decrypt the symmetric encryption key. Again, you already know how this works because it is identical to the rsautl example above:

openssl rsautl -decrypt -inkey private-key.pem < test-encrypted.txt > test.txt

test.txt now contains “foobar” and we can use this to decrypt the word list:

openssl enc -aes-128-cbc -d -pass file:test.txt < words.dat > words.txt

You might be wondering why we don’t simply use the symmetric algorithm directly to exchange data, instead of going to such lengths to combine the two methods. The reason is that asymmetric keys have desirable properties regarding key distribution and ownership, especially when encryption is used on a large scale, such as between millions of users on the Internet. The Wikipedia article linked above has more information.

Signing and Verifying

Here’s a look at the other operation we can perform with a key pair, signing and verifying messages. As noted above, this lets the recipient determine if the message has been altered in transit.

As with encryption, sender and recipient must have exchanged their respective public keys before they can exchange signed messages. In this use case, the sender uses his or her own private key to sign the message.

Signing is actually encryption with reversed keys; encryption is performed using the private key and decryption using the public key. Here’s an example of this fact, with some help from Alice again:

echo "Alice in Wonderland" > test.txt

rsautl is also the subcommand for signing and verifying:

openssl rsautl -sign -inkey private-key.pem < test.txt > signed.dat

signed.dat now contains the signature data, which is the message data “Alice in Wonderland” encrypted with the private key. It’s unreadable binary data.

Anyone with the public key can decrypt this data with rsautl’s -verify option:

openssl rsautl -verify -inkey public-key.pem -pubin < signed.dat

This yields:

Alice in Wonderland

Now let’s consider how this helps us. Using the keys this way means that everyone can read the message, as the public key is, well, public. So how can this possibly be useful?

It’s useful because the recipient being able to successfully decrypt the message using the sender’s public key also means something else: It means that the message in its decrypted form must have been sent by the sender who gave us the public key (because only the sender has the related private key), and the message must have been sent exactly the way it came out of the decryption process. It tells us unambiguously that the message was sent by who we think sent it, and that it has arrived exactly the way it was sent.

You saw during the discussion of encryption/decryption that the amount of data that can be processed directly with the asymmetric algorithm is limited, and our little “Alice in Wonderland” message was short enough but most real-world messages are not. If we want to transmit our English word list as a signed message, it won’t work because the data is too big:

openssl rsautl -sign -inkey private-key.pem < /usr/share/dict/words > signed.dat

RSA operation error
79490:error:0406C06E:rsa routines:RSA_padding_add_PKCS1_type_1:data too large for key size:rsa_pk1.c:73:

Like with encryption/decryption, we can use a clever combination of the public key algorithms with a second technique to solve this problem. In this case the second technique is a cryptographic hash function. If you don’t know what a hash function is, take a look at the Wikipedia description first. It says:

A cryptographic hash function is a deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string, the hash value, such that an accidental or intentional change to the data will almost certainly change the hash value. In many contexts, especially telecommunications, the data to be encoded is often called the "message", and the hash value is also called the message digest or simply digest.

The ideal hash function has four main properties: it is easy to compute the hash for any given data, it is extremely difficult to construct a text that has a given hash, it is extremely difficult to modify a given text without changing its hash, and it is extremely unlikely that two different messages will have the same hash. These requirements call for the use of advanced cryptography techniques, hence the name.

Our English word list is the “arbitrary block of data” mentioned in the description.

openssl has the dgst subcommand for cryptographic hash functions. We choose the SHA1 algorithm:

openssl dgst -sha1 < /usr/share/dict/words

The resulting digest:

a7d455ffb810853b51dfd3613af24333ecc603e9

Adding (or changing) just a single letter to the word list results in a completely different digest, clearly indicating that the input data has changed:

echo -n x >> /usr/share/dict/words
openssl dgst -sha1 < /usr/share/dict/words

20b920565a9351b259deb11acd6e12867dec8240

The message digest is always of the same fixed length, 20 bytes in the case of SHA1. This is short enough for the public key algorithms. This means that we now have all the pieces in place to sign arbitrarily large messages.

The sender of the message begins by calculating the digest for the message data:

openssl dgst -sha1 < /usr/share/dict/words > digest.txt

The sender signs the digest with the private key:

openssl rsautl -sign -inkey private-key.pem < digest.txt > digest-signed.dat

The signed message to the recipient consists of two parts: The message payload, in this example the /usr/share/dict/words file containing the word list, and the signed digest, digest-signed.dat.

The recipient first reverses the signing operation, which if successful yields the digest of the payload data as determined by the sender:

openssl rsautl -verify -inkey public-key.pem -pubin < digest-signed.dat > digest-sender.txt

If this doesn’t work, the recipient already knows that something is wrong.

Otherwise, the next step is to calculate the message digest of the message data, exactly the same way the sender did it:

openssl dgst -sha1 < words > digest-local.txt

If the two hash values in digest-sender.txt and digest-local.txt are identical, the recipient knows that the message arrived exactly as the sender sent it. If they differ, the message was altered in transit:

diff digest-{sender,local}.txt

1c1
< a7d455ffb810853b51dfd3613af24333ecc603e9
---
> d0b081cb449533c4c9ad83595c1b35a17f12e625

Note: In my experience, it’s often a source of confusion that the key types are reversed for encryption/decryption (sender uses a public key, recipient uses a private key) and signing/verifying (sender uses a private key, recipient uses a public key), so think about this for a moment to keep it straight in your head...

As a reward for making it all the way through this part, here’s a comic for you :-)

What’s next

This part introduced the tools needed to exchange encrypted and signed messages. The next part will show the practical problems of public key exchange/distribution and how a Public Key Infrastructure helps with that problem.

Recent versions of the Sparkle auto-update framework used by many Mac OS X applications require the publisher to protect their users from malicious attacks by either delivering application updates over SSL or digitally signing them, which is a good thing.

Signing involves a few steps with OpenSSL that need to be executed again for every release build. To make this more reliable and convenient, I automated the process with a custom script phase in my “distribution” Xcode target.

The first step is to generate a DSA key pair as documented on the Sparkle website. This gives you the private key in dsa_priv.pem and the public key in dsa_pub.pem.

The private key is sensitive and I did not want it lying around on the file system, so I added it to my keychain as a Secure Note in the Keychain Access utility:

For completeness, I also added the public key even though it is not sensitive, resulting in these two secure note items:

I deleted the private key from the file system, moved the public key to the resources of my Xcode project and included its name in the Info.plist file under the SUPublicDSAKeyFile key, again as documented on the Sparkle Website.

In Xcode, I added a “Distribution” shell script target:

This is the shell script code:

set -o errexit

[ $BUILD_STYLE = Release ] || { echo Distribution target requires "'Release'" build style; false; }

VERSION=$(defaults read "$BUILT_PRODUCTS_DIR/$PROJECT_NAME.app/Contents/Info" CFBundleVersion)
DOWNLOAD_BASE_URL="http://www.example.com/download/"
RELEASENOTES_URL="http://www.example.com/software/my-cool-app/release-notes.html#version-$VERSION"

ARCHIVE_FILENAME="$PROJECT_NAME $VERSION.zip"
DOWNLOAD_URL="$DOWNLOAD_BASE_URL/$ARCHIVE_FILENAME"
KEYCHAIN_PRIVKEY_NAME="Sparkle Private Key 1"

WD=$PWD
cd "$BUILT_PRODUCTS_DIR"
rm -f "$PROJECT_NAME"*.zip
ditto -ck --keepParent "$PROJECT_NAME.app" "$ARCHIVE_FILENAME"

SIZE=$(stat -f %z "$ARCHIVE_FILENAME")
PUBDATE=$(date +"%a, %d %b %G %T %z")
SIGNATURE=$(
	openssl dgst -sha1 -binary < "$ARCHIVE_FILENAME" \
	| openssl dgst -dss1 -sign <(security find-generic-password -g -s "$KEYCHAIN_PRIVKEY_NAME" 2>&1 1>/dev/null | perl -pe '($_) = /"(.+)"/; s/\\012/\n/g') \
	| openssl enc -base64
)

[ $SIGNATURE ] || { echo Unable to load signing private key with name "'$KEYCHAIN_PRIVKEY_NAME'" from keychain; false; }

cat <<EOF
		<item>
			<title>Version $VERSION</title>
			<sparkle:releaseNotesLink>$RELEASENOTES_URL</sparkle:releaseNotesLink>
			<pubDate>$PUBDATE</pubDate>
			<enclosure
				url="$DOWNLOAD_URL"
				sparkle:version="$VERSION"
				type="application/octet-stream"
				length="$SIZE"
				sparkle:dsaSignature="$SIGNATURE"
			/>
		</item>
EOF

echo scp "'$HOME/svn/my-cool-app/build/Release/$ARCHIVE_FILENAME'" www.example.com:download/
echo scp "'$WD/appcast.xml'" www.example.com:web/software/my-cool-app/appcast.xml

You have to set the shell to /bin/bash because this code uses features that are not available otherwise.

This script:

• enforces a Release build
• creates a .zip file of the application. You could also package up a .dmg with additional material.
• fetches the signing private key from the keychain. This step might pop up a dialog:
• calculates the SHA1 checksum of the distribution file
• signs the checksum with the private key
• converts the signature to Base64
• emits an item block with all the information (date, size, version) about the update. You add this block to your appcast XML file.

I also let it print out the scp commands required to publish the update.

Update: Alan Craig is working on an extended Ruby version of the bash script, featuring a few more options.

Snow Leopard Update: It seems that on Snow Leopard, the security tool’s output changed. You have to replace the multiline variable assignment to SIGNATURE above with this:

SIGNATURE=$(
	openssl dgst -sha1 -binary < "$ARCHIVE_FILENAME" \
	| openssl dgst -dss1 -sign <(security find-generic-password -g -s "$KEYCHAIN_PRIVKEY_NAME" 2>&1 1>/dev/null | perl -pe '($_) = /"(.+)"/; s/\\012/\n/g' | perl -MXML::LibXML -e 'print XML::LibXML->new()->parse_file("-")->findvalue(q(//string[preceding-sibling::key[1] = "NOTE"]))') \
	| openssl enc -base64
)

This is because the output is now an XML property list with multiple items.

Update: An earlier version of this article used this command to produce the distribution ZIP file:

zip -qr "$ARCHIVE_FILENAME" "$PROJECT_NAME.app"

It turns out that this compresses inefficiently. Thanks to Fabian Jäger for the ditto line used above, which compresses more efficiently and produces smaller files.

When I work on Cocoa programs I read the framework documentation with Xcode’s built in documentation viewer, but sometimes I still open the HTML page for a particular class in Safari with LaunchBar because it is very quick (see this older post).

Xcode downloads these documentation sets from the Internet, and unfortunately all of the thousands of files in there are tagged with the “quarantine” extended file system attribute. This means that the system warns you about potentially harmful content every time you open one of the .html files outside of Xcode, such as in the LaunchBar case.

To get around this, you can clear the quarantine attribute on all files with this command:

find /Developer/Documentation -type f -print0 | xargs -n 100 -0 sudo xattr -d com.apple.quarantine