Signing an EC2 API request in Python

My last post was about a simple script I wrote to start up an EC2 instance and verify the SSH fingerprint. It did the trick but left me itching for something that was a little easier to extend and generalize.

I've also been itching to learn some Python, so for my next experiment I wrote a Python script (my first) that generates the security signature for an EC2 API request. Google pointed me to several solutions, but they were all either outdated (version 1 of the signature specification), incomplete, inaccurate or all of the above.

The following script works for me. It is still hard coded to perform a single type of simple request, but it should be a useful example for anybody that just wants to sign an EC2 request in Python.

The boto project also provides what looks like a fairly complete Python interface to most of the Amazon web services.

#!/usr/bin/python3

import base64,hashlib,hmac,datetime,urllib.parse,urllib.request

endpoint = 'ec2.us-west-1.amazonaws.com'
aws_secret_access_key = b'YOUR_KEY_SECRET'

params = {}
params['Version'] = '2011-01-01'
params['AWSAccessKeyId'] = 'YOUR_ACCESS_KEY'

now = datetime.datetime.utcnow()
params['Timestamp'] = now.strftime("%Y-%m-%dT%H:%M:%S.000Z")

# this can be omitted
expires = now + datetime.timedelta(minutes=10)
params['Expires'] = expires.strftime("%Y-%m-%dT%H:%M:%SZ")

params['SignatureMethod'] = 'HmacSHA256'
params['SignatureVersion'] = '2'

params['Action'] = 'DescribeInstances'

keys = sorted(params.keys())
values = map(params.get, keys)

query_string = urllib.parse.urlencode( list(zip(keys,values)) )

# construct the string to sign
string_to_sign = '\n'.join(['GET', endpoint, '/', query_string])

# sign the request
signature = hmac.new(
    key=aws_secret_access_key,
    msg=bytes(string_to_sign, 'UTF-8'),
    digestmod=hashlib.sha256).digest()
 
# base64 encode the signature.  URL safe base64 encoding does NOT work 
signature = base64.b64encode(signature)

signature = urllib.parse.quote(signature) 

url = 'https://' + endpoint + '/?' + query_string + '&Signature=' + signature

response = urllib.request.urlopen(url)
xml = response.read()

# learning a little about xml parsing in Python is my next project
print(xml)

Starting and Verifying SSH Fingerprint of an EC2 instance

Lately I've been doing some experimentation on a temporary Amazon EC2 instance. So far I have been starting each session by running ec2-run-instances on the command line to boot the instance, then running ec2-describe-instances to get the host name, and finally connecting via SSH.

Scott Moser's post inspired me to automate the process as well as verify the SSH fingerprint of the new instance:

#!/bin/sh

# start an EC2 micro instance running the 32-bit, EBS-backed Amazon Linux AMI
RUN=`ec2-run-instances ami-3bc9997e -k KEY -t t1.micro -z us-west-1c`

# retrieve the instance ID from the output
INSTANCE=`echo $RUN | grep -E -o ' i-[a-f0-9]+' | sed 's/INSTANCE  *//'`

# seems to take about 3 to 4 minutes for SSH fingerprints to show
# up in the output. wait for 2 and a half minutes, then start polling output
echo "Waiting 150s for $INSTANCE to boot"

sleep 150

while [ 1 ]
do

FINGERPRINTS=`ec2-get-console-output $INSTANCE | egrep -m 1 -o '([0-9a-f][0-9a-f]:){15}[0-9a-f][0-9a-f]'`

if [ "$FINGERPRINTS" = "" ]
then
  sleep 30
  echo "Booting..."
else
  break
fi

done

echo "Expected fingerprints are $FINGERPRINTS"

# get hostname for the instance
HOST=`ec2-describe-instances | grep -m 1 $INSTANCE | egrep -o 'ec2(-[0-9]+){4}.us-west-1.compute.amazonaws.com'`

echo "Host is $HOST"

ssh-keyscan $HOST 2>/dev/null > host.key

ssh-keygen -lf host.key > host.fingerprint

read len ACTUAL_FINGERPRINTS host rsa < host.fingerprint
echo "Actual fingerprints are $ACTUAL_FINGERPRINTS"

if [ "$ACTUAL_FINGERPRINTS" = "$FINGERPRINTS" ]
then

echo "Fingerprints match, adding to known hosts"

ssh-keygen -q -R "$HOST"

ssh-keygen -q -H -f host.key

cat host.key >> ~/.ssh/known_hosts

echo "Ready to connect"

echo "ssh -i PATH_TO_KEY ec2-user@$HOST"

else

echo "Fingerprints do not match"

fi

shred -u host.key host.fingerprint

Thanks Scott!

Optimizing GWT for iPhone

I have been working sporadically on my math quiz application for the past several weeks. To complicate matters beyond just learning GWT, I decided to optimize the user interface for the iPhone. That way my daughter could practice her math on the go.

First of all, here is the current version of the application, deployed on Google App Engine:

http://1.latest.mookiemath.appspot.com/

Notice a few things:

1. After you login, the application shows you the quiz screen directly.
2. You can click or tap on the buttons at the bottom of the screen to answer the problems.
3. If you have a keyboard, you can also use the corresponding number keys (capturing the browser key press events is for another post)
4. After you complete six problems, the application saves the result to the server.
5. You can click on the little door icon to switch to a history view that lists your quiz results.

Overall, a very minimalist interface designed for a small screen.

Once I mastered how to apply styles to widgets, there were only two specific tweaks I had to make for the iPhone:

Scaling

Since this is a full-screen application rather than a web page, it needs to disable scaling by adding this meta tag to host page header:

<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0;">

Scrolling

The history view consists of a fixed header and a scrollable body:

  <g:DockLayoutPanel unit="EM">
    <g:north size="3.2">
      <g:FlowPanel>
        <g:Label ui:field="iphoneHelp">
          Use two fingers to scroll
        </g:Label>
        <g:Button ui:field="quizButton" text="Quiz"/>
      </g:FlowPanel>
    </g:north>
    <g:center>
      <g:ScrollPanel>
        <g:FlowPanel ui:field="history" />
      </g:ScrollPanel>
    </g:center>
  </g:DockLayoutPanel>

A ScrollPanel is basically just a DIV with the CSS overflow property set to auto. This works fine in a full-sized browser, but mobile Safari does not respect the auto setting by showing a scrollbar when the content overflows. Instead, you have to use a two-finger gesture to scroll the content.

I didn't know this until today, and I expect that many other people do not either, so I wanted to add a simple help label that would only appear for iPhone users. GWT provides "deferred binding" as a general technique for substituting client classes (including widgets) based on user agent or other properties, but that seemed like overkill for this one label. Instead I just relied on a simple native method in my HistoryPanel widget to retrieve the user agent and hide the help label for non-iPhone users:

public class HistoryPanel extends Composite {

  ...

  @UiField
  Label iphoneHelp;

  public HistoryPanel(final HandlerManager handlerManager) {

    DockLayoutPanel parent = uiBinder.createAndBindUi(this);
    initWidget(parent);

    if (getUserAgent().indexOf("iphone") == -1) {
      iphoneHelp.removeFromParent();
    }

    ...
  }

  private native String getUserAgent() /*-{
    return navigator.userAgent.toLowerCase();
  }-*/;

  ...
}

Other than those two issues, the layout looks surprisingly consistent on full-sized browsers and the iPhone.

Dynamic Layouts in GWT

After figuring out how to identify the current user, the next task in building my math quiz application was to build a basic layout for the quiz itself.

For starters, all I need is a header that displays the name of the current user, and a content area that displays the math problems. A user answers a problem by filling in a text box and hitting enter, after which the program should refresh the content area and show a new problem. After they've done a certain number of problems, the content area shows an "All done" message.

Implementing the basic layout and associated dynamic logic was fairly simple. First I added a body area to the layout:

<ui:UiBinder xmlns:ui='urn:ui:com.google.gwt.uibinder'
  xmlns:g='urn:import:com.google.gwt.user.client.ui'>

  <g:DockLayoutPanel unit="EM">

    <g:north size="4">
      <g:HTML>
        Hello,
        <span ui:field='nameSpan' />
      </g:HTML>
    </g:north>

    <g:center>
      <g:FlowPanel ui:field="problemPanel">
      </g:FlowPanel>
    </g:center>

  </g:DockLayoutPanel>

</ui:UiBinder>

Then I added a method to my EntryPoint implementation, which clears the problemPanel and shows a new problem up to a fixed limit:

public class Uitest implements EntryPoint {

  ...

  private int problemCount = 0;

  public void onModuleLoad() {

    ...
    showProblem();
  }
  
  private void showProblem() {
    
    problemPanel.clear();
    
    long a = Math.round(Math.random() * 10);
    long b = Math.round(Math.random() * 10);

    problemPanel.add(new InlineLabel(Long.toString(a)));
    problemPanel.add(new InlineLabel("+"));
    problemPanel.add(new InlineLabel(Long.toString(b)));
    problemPanel.add(new InlineLabel("="));
    
    final TextBox textBox = new TextBox();
    textBox.setWidth("2em");
    
    textBox.addChangeHandler(new ChangeHandler() {

      @Override
      public void onChange(ChangeEvent event) {
        if (problemCount++ < 5) {
          showProblem();
        } else {
          problemPanel.clear();
          problemPanel.add(new Label("All done!"));
        }
      }
    });
    
    problemPanel.add(textBox);

    textBox.setFocus(true);    
  }
}

I am assuming that it is acceptable for the class to be stateful (keep track of the count of problems for a single user), since this code runs on the client.

I ran the project in Eclipse and it worked as expected...almost. The positioning of the problem was oddly shifted on first viewing the page, and then it seemed to straighten itself out after the first answer. By process of elimination I discovered that the setFocus was causing this. First I verified that this method was not changing the DOM in any way; it was not. After looking through the docs I finally tried deferring the focus command until after all other pending operations were complete:

DeferredCommand.addCommand(new Command() {

      @Override
      public void execute() {
        textBox.setFocus(true);
      }
    });

This solved the display problem. Now I was ready to implement the remote service to record the user's score.

Thinking in GWT

The next task in building my math quiz application was to figure out how to identify visitors so I can track their quiz points. I figure most people either have a Google account or don't mind getting one, so I can just leverage the Google Accounts service that is provided with App Engine.

First I added the security constraint to web.xml:

<security-constraint>
  <web-resource-collection>
      <url-pattern>/Uitest.html</url-pattern>
  </web-resource-collection>
  <auth-constraint>
      <role-name>*</role-name>
  </auth-constraint>
</security-constraint>

After adding that and restarting my dev server, the application redirected to the dummy login page as expected.

The next challenge was to display the identity of the user on the page. Following one of the sample programs, I updated the template to include a dynamic field:

<ui:UiBinder xmlns:ui='urn:ui:com.google.gwt.uibinder'
  xmlns:g='urn:import:com.google.gwt.user.client.ui'>

  <g:DockLayoutPanel unit='EM'>

    <g:center>
      <g:HTML>
        <p>Hello, <span ui:field='nameSpan'/>.</p>
      </g:HTML>
    </g:center>

  </g:DockLayoutPanel>

</ui:UiBinder>

Then I hit the wall. My instinct from years of writing servlets and JSPs was to get the user principal from the request and use it to set the value of the nameSpan field. But none of the documentation and samples and examples I could find on the web seemed to show how to do that.

Unsure how to proceed, I created a new project and took another look at the GreetingService that the GWT plugin generates. I noticed that the server implementation of the service extends RemoteServiceServlet, which provides a method for accessing the current request. At first I wasn't sure how I was going to use it, but I modified the server implementation to simply return the current user:

public class GreetingServiceImpl extends RemoteServiceServlet implements
  GreetingService {

  public String greeting() throws IllegalArgumentException {

    return getThreadLocalRequest().getUserPrincipal().getName();
  }
}

Then I updated the client interface accordingly:

@RemoteServiceRelativePath("greet")
public interface GreetingService extends RemoteService {
  String greeting() throws IllegalArgumentException;
}

and, following the errors reported by Eclipse, lastly I updated the async version of the interface:

public interface GreetingServiceAsync {
  void greeting(AsyncCallback callback)
      throws IllegalArgumentException;
}

So now I had a service that returns the current user name, but I wasn't sure how to use it yet. After looking at the samples a bit more it finally dawned on me that I needed to change my whole perspective. Writing Java code no longer automatically means that the code is executing on the server. GWT "compiles" the Java code into JavaScript to run on the client. Admittedly, this is all explained well in the documentation but it had been a while since I read it.

Any information that I needed to retrieve from the server needed to be obtained via a callback:

public class Uitest implements EntryPoint {

  interface Binder extends UiBinder {
  }

  private static final Binder binder = GWT.create(Binder.class);

  private final GreetingServiceAsync greetingService =
    GWT.create(GreetingService.class);

  @UiField
  SpanElement nameSpan;

  public void onModuleLoad() {

    DockLayoutPanel outer = binder.createAndBindUi(this);

    Window.enableScrolling(false);
    Window.setMargin("0px");

    RootLayoutPanel root = RootLayoutPanel.get();
    root.add(outer);

    greetingService.greeting(new AsyncCallback() {

      @Override
      public void onSuccess(String result) {
        nameSpan.setInnerText(result);
      }

      @Override
      public void onFailure(Throwable caught) {
        // TODO implement a dialog box to show error
        nameSpan.setInnerText("ERROR");
      }
    });
  }
}

This didn't work the first time I tried to run the application, but then I realized that I needed to add the Greeting servlet to web.xml:

  <servlet>
    <servlet-name>greetServlet</servlet-name>
    <servlet-class>org.sample.uitest.server.GreetingServiceImpl</servlet-class>
  </servlet>

  <servlet-mapping>
    <servlet-name>greetServlet</servlet-name>
    <url-pattern>/uitest/greet</url-pattern>
  </servlet-mapping>

Now I was able to run the application and see the user name show up.

The simplest possible GWT application using UiBinder

I've been thinking about building a application to help my daughters practice math. The idea is to generate a simple daily quiz that they take online. The application should send them an SMS or email reminder to take the quiz, and then track their results. Each completed quiz should earn them points (number of correct answered problems) that they can redeem with me (in exchange for something to-be-decided). Each day they fail to get over some threshold or fail to take the quiz at all, they lose some number of number of points.

Google App Engine seems like a good fit for implementing this app, and anyway I've been wanting to try it out and also learn how to use GWT. I just installed the latest Eclipse plugin (App Engine 1.3.1 and GWT 2.0.2).

Where to begin? Whenever I learn a new technology, I find it helpful to start with the simplest possible scenario and then iterate, making sure I understand each additional layer of complexity. A logical starting point was just to build an app that outputs static HTML.

GWT 2.0 introduced a new declarative technique for defining the UI in XML markup rather than code. The UiBinder documentation has examples but doesn't have a complete tutorial that assumes no previous knowledge of GWT. I started with the Mail sample app and pared it down to the simplest possible example.

The first step was to create a new Web Application Project in Eclipse using org.sample.uitest as the default project package. Then I deleted all of the boilerplate code that gets generated for the new project, leaving a src directory with empty client, server and shared packages. I also removed the servlet mapping from war/WEB-INF/web.xml.

The first file that was clearly necessary was the 'module definition' in src/org/sample/uites/Uitest.gwt.xml. The simplest possible module definition inherits the basic toolkit functionality and stylesheet and defines an entry point into the application:

<module rename-to='uitest'>

  <inherits name='com.google.gwt.user.User'/>

  <inherits name='com.google.gwt.user.theme.standard.Standard'/>

  <entry-point class='org.sample.uitest.client.Uitest'/>

</module>

Next I needed to implement the entry point class. I started by copying the entry point from the Mail sample app and paring it down:

package org.sample.uitest.client;

import com.google.gwt.core.client.EntryPoint;
import com.google.gwt.core.client.GWT;
import com.google.gwt.uibinder.client.UiBinder;
import com.google.gwt.user.client.Window;
import com.google.gwt.user.client.ui.DockLayoutPanel;
import com.google.gwt.user.client.ui.RootLayoutPanel;

public class Uitest implements EntryPoint {

  interface Binder extends UiBinder { }

  private static final Binder binder = GWT.create(Binder.class);

  public void onModuleLoad() {

    DockLayoutPanel outer = binder.createAndBindUi(this);

    Window.enableScrolling(false);
    Window.setMargin("0px");

    RootLayoutPanel root = RootLayoutPanel.get();
    root.add(outer);
  }
}

The Binder requires a matching layout file named Uitest.ui.xml:

<ui:UiBinder
  xmlns:ui='urn:ui:com.google.gwt.uibinder'
  xmlns:g='urn:import:com.google.gwt.user.client.ui'>

  <g:DockLayoutPanel unit='EM'>

    <g:center>
      <g:HTML>
        <p>UI Test</p>
      </g:HTML>
 
    </g:center>

  </g:DockLayoutPanel>

</ui:UiBinder>

Finally, an HTML page named Uitest.html is required in the war directory to 'host' the application:

<!doctype html>

<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <title>UI Test</title>
    <script type="text/javascript" language='javascript' src='uitest/uitest.nocache.js'></script>
  </head>
  <body>
    <noscript>
      <div style="width: 22em; position: absolute; left: 50%; margin-left: -11em; color: red; background-color:white; border: 1px solid red; padding: 4px; font-family: sans-serif">
        Your web browser must have JavaScript enabled
        in order for this application to display correctly.
      </div>
    </noscript>
  </body>
</html>

At this point my project consisted of five files, including WEB-INF/web.xml. I ran the project in Eclipse and was able to see the message in the browser.

Out of curiosity, I looked at the requests in Firebug when the page loads. The initial response is simply an unmodified version of the 'host' page, which loads uitest.nocache.js to actually render the page. It appears that uitest.nocache.js executes an onload handler that in turn triggers a request for hosted.html?uitest. I didn't try very hard to decipher the obfuscated JavaScript, but it appears that it is writing a SCRIPT tag to load the dynamic content.

Log4J and Spring

I just spent a half hour trying to increase the logging for a unit test that uses SpringJUnit4ClassRunner. First I thought I could just add a log4j.xml configuration file at the root of the classpath, but that didn't work. After looking around the Spring code a bit it dawned on me that I didn't even have log4j on the classpath for the project, so Commons Logging was using the standard java.util.logging package instead. I added log4j as a dependency to the project, and now log4j configuration works as expected.