Blog has moved to https://jamiecraane.dev/

My personal blog has moved to https://jamiecraane.dev/

AWS tutorial: Retrieve items from DynamoDB using Lambda and API Gateway

Introduction

Part 1: http://jcraane.blogspot.nl/2016/12/aws-lambdajava-dynamodb-and-api-gateway.html

source code: https://github.com/jcraane/LambdaDynamoDBApiGateway

In the previous tutorial I showed you how to use AWS Lambda and API Gateway to insert items in a DynamoDB table. The example implemented a function which stored the location of the user. In this tutorial we create a Lambda function which retrieves this data from the DynamoDB table and expose this functionality over HTTP using API Gateway.

Create a DynamoDB Global Secondary Index

The DeviceLocation DynamoDB table uses the id as partition key. To retrieve the locations for a given device, a global secondary index must be created to query on deviceId.

1. Open the AWS console and navigate to the DynamoDB section.

2. Select the DeviceLocation table and click the Indexes tab.

3. Select Create Index.

4. Use deviceId as the partition key, use 1 for read/write capacity units and click create. See the create_dynamodb_index screenshot.

create_dynamodb_index

Create the implementation

1. Create a new class named RetrieveLocationRequest with one field named “deviceId” with type String. This class holds the parameters which are used as query input to the  DynamoDB table.

2. Create a new class named RetrieveLocationResponse with a field named “locations” of type List. This class will hold the data returned to the consumer of the Lambda function.

3. Initialize the locations field to: new ArrayList<>() since we are going to add DeviceLocation objects to it.

4. Create a new class named RetrieveLocationFunction which implements the com.amazonaws.services.lambda.runtime.RequestHandler interface.

The RetrieveLocationFunction is the implementation of our Lambda function which queries DynamoDB using the given deviceId. It retrieves all locations for the given device.

Copy the following code and paste this in the handleRequest method of the RetrieveLocationFunction class. Make sure the region used here is the same region as the region of the DynamoDB table!

final AmazonDynamoDBClient client = new AmazonDynamoDBClient(new EnvironmentVariableCredentialsProvider());

client.withRegion(Regions.EU_CENTRAL_1); // specify the region you created the table in.

final DynamoDB dynamoDB = new DynamoDB(client);

System.out.println("input = " + input); // Pure for testing. Do not use System.out in production code

final Table table = dynamoDB.getTable("DeviceLocation");

final Index index = table.getIndex("deviceId-index");

final ItemCollection items = index.query("deviceId", input.getDeviceId());

final RetrieveLocationResponse response = new RetrieveLocationResponse();

for (final Item item : items) {

    final DeviceLocation deviceLocation = new DeviceLocation();

    deviceLocation.setDeviceId(item.getString("deviceId"));

    deviceLocation.setLat(item.getDouble("lat"));

    deviceLocation.setLng(item.getDouble("lng"));

    response.getLocations().add(deviceLocation);

}

return response;

Create the Lambda function

After the implementation is ready we need to upload the function and configure it in AWS.

1. Run the Gradle “clean build” task to create the zip distribution which holds our code and dependencies.

2. Open the AWS console and navigate to the Lambda section.

3. Select “Create a Lambda function” and select Blank Function.

4. Click Next.

5. As name specify “retrieveLocations” and as runtime select Java 8.

6. Upload the PROJECT/build/distributions/DISTRIBUTION_NAME.zip file

7. Use com.example.persister.RetrieveLocationFunction as the Handler

8. Use the existing lambda_location_persister role create in part 1 of this series.

9. Leave the rest default and click Next.

9. Click Create Function.

10. To test the function click Test and use the following test data (replace DEVICE_ID with the deviceId of a record in the table. Refer to part 1 to insert data in the DeviceLocation table):

{

  "deviceId": “DEVICE_ID”

}

Exposing the functionality using API gateway

The retrieveLocations function will be exposed using an HTTP GET method using API Gateway.

1. Open the AWS console and navigate to API Gateway.

2. Create a new GET method under the deviceLocation resource.

3. As Integration Type choose Lambda Function and select the region the Lambda Function was created in.

4. Specify retrieveLocations as the name and click Save.

The deviceId is passed in as a query parameter in the URL and must be mapped as input to the Lambda function.

1. Open the Method Request settings and open URL Query String Parameters.

2. Add a query parameter named deviceId.

3. Open the Integration Request settings and open Body Mapping Templates.

4. Choose the option: When there are no templates defined (recommended)

5. Add a mapping template for: application/json

6. Add the following template to map the deviceId URL query parameter to the Lambda input (see screenshot body_mapping).

{

    "deviceId": "$input.params('deviceId')"

}

body_mapping

7. Click Save.

8. Click Test, add a known deviceId and Click the Test button. You should see the output of the Lambda function.

9. Click Actions -> Deploy API to deploy your api.

10. You function should now be publicly accessible.

Make sure you delete your resources when you are done with the tutorial to prevent unwanted billing.

AWS Lambda/Java, DynamoDB and Api gateway integration

Introduction

Part 2: http://jcraane.blogspot.nl/2016/12/aws-tutorial-retrieve-items-from.html

In this post I am going through a full (Java) example of integrating AWS Lambda, DynamoDb and Api Gateway to create a function and expose this function as a HTTP resource for other parties to consume.

Before we dive into the details I will give a brief overview of the AWS services used in this example (as taken from the AWS documentation):

• AWS Lambda. AWS Lambda is a compute service that runs developers' code in response to events and automatically manages the compute resources for them, making it easy to build applications that respond quickly to new information.
• DynamoDB: Fast and flexible, managed, NoSql database.
• Api Gateway: Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.

In this example we are going to create a lambda function which tracks the location (latitude and longitude) of a specific mobile device. The data flow looks like this:

mobile device -> HTTP POST -> Api Gateway -> recordLocation (Lambda function) -> DynamoDb (store location)

Creating the application

Prerequisites:

• IntelliJ Idea is used for this example but any IDE will do. Gradle is used for the build system.
• AWS account to actually deploy and run the example.
• After you are done with the example, delete any AWS resources you have created to prevent unnecessary billing.

For the implementation of the Lambda function we use the AWS Java SDK.

1. In IntelliJ, Select File -> New project and choose Gradle with the Java library. Click Next
2. groupId=com.example, artifactId=locationpersister. Click Next.
3. Choose a Java 8 IDE and Click Next, click Finish.
4. Create the src/main/java folder if it does not exist already.
5. Open de build.gradle file and add the following dependencies:

compile 'com.amazonaws:aws-lambda-java-core:1.1.0'
compile 'com.amazonaws:aws-lambda-java-events:1.1.0'

6. Add the following code to the build.gradle file:

task buildZip(type: Zip) {
    from compileJava
    from processResources
    into('lib') {
        from configurations.runtime
    }
}
build.dependsOn buildZip

The above code creates a zip archive when the build task is triggered. The zip file can be uploaded directly to AWS Lambda.

7. Create a class com.example.persister.DeviceLocation with the members: lat (double), lng (double) and deviceId (string). This class holds the data that gets submitted to the Lambda function.
8. Create a new class com.example.persister.LocationPersisterFunction. This class will hold the implementation of the Lambda function.
9. Make the LocationPersisterFunction implement the com.amazonaws.services.lambda.runtime.RequestHandler interface.
This interface defines the handleRequest function which is executed when the Lambda function is triggered. The handleRequest function takes two parameters: the input (which is of DeviceLocation type and is passed-in when the function is invoked Context object).

Creating the DynamoDb table

To store the data in DynamoDB we need to create a table.

1. Open the AWS console and navigate to the DynamoDB section.
2. Click create Table and for the name use: "DeviceLocation". Type id (string field) to use as a partition key and do not specify a sort key. Leave everything default and click Create.
3. Please note that the table is created in the selected regio. If you click on the table and look at the details you can see the region of the table.

Create the DeviceLocation class

The DeviceLocation class holds the input which is passed to the Lambda function.
1. Create a new class named DeviceLocation with the following fields: id (string), lat (double) and lng (double)
2. Make sure the class contains both setters and getters. The setters are used by AWS Lambda to populate this object based on the passed in JSON when calling the Lambda function.

Implementing the Lambda function

1. Open the LocationPersisterFunction
2. Add the following code to the body of the handleRequest method:

final AmazonDynamoDBClient client = new AmazonDynamoDBClient(new EnvironmentVariableCredentialsProvider());
        client.withRegion(Regions.EU_WEST_1); // specify the region you created the table in.
        DynamoDB dynamoDB = new DynamoDB(client);
        Table table = dynamoDB.getTable("DeviceLocation");
        final Item item = new Item()
                .withPrimaryKey("id", UUID.randomUUID().toString()) // Every item gets a unique id
                .withString("deviceId", input.getDeviceId())
                .withDouble("lat", input.getLat())
                .withDouble("lng", input.getLng());
        table.putItem(item);
        return null;

3. Make sure you change the region to match the region you created the table in.
4. The above code gets a reference to the DynamoDB DeviceLocation table, creates an item and persist it.
5. Execute the gradle build task to create a zip-archive or our code.
5. Now that the implementation is complete we are ready to create our AWS Lambda function.

Creating out Lambda function

1. Open the AWS console and navigate to the Lambda section.
2. Select Blank function and click next (we do not create a trigger at this stage).
3. As name choose: persistDeviceLocation and select Java 8 as the runtime
4. Upload the /build/distributions/locationpersister-1.0-SNAPSHOT.zip file
5. In the Handler field specify the fully qualified classname which implements our handler: com.example.persister.LocationPersisterFunction
6. In the Role field select to create a custom role. The create role form is opened. Use lambda_location_persister as the Role name and click allow. The role is created and selected in the Existing Role field. See the screenshot "lambda_role"
7. Leave everything default and click Next
8. Click Create function

lambda_role

Testing the function in de AWS console

After the function is created we are going to test is using the AWS console.
1. Click Test
2. A dialog opens where you can specify the data sent to the Lambda function. Use the following testdata:

{
  "deviceId": "deviceId",
  "lat": 52.5,
  "lng": 5.5
}

3. You can modify the testdata at any time by clicking Actions -> configure test event
4. When done, click Test
5. If everything went according to plan you should get an error message which states the following: Status Code: 400; Error Code: AccessDeniedException
6. This is correct until this point. Although we created a custom role, we did not gave this role permissions to access our DynamoDB table.

Add DynamoDB permissions to our role

1. Open the AWS console and navigate to the IAM section.
2. Click on Roles
3. Click on the lambda_location_persister to open it.
4. Click Attach Policy
5. In the filter field search for DynamoDB
6. Select the AmazonDynamoDBFullAccess policy and click Attach
7. Navigate back to AWS Lambda and test the function again. The function should be succesfull.
8. Navigate to DynamoDB and select the DeviceLocation table and click on items. You should see one item added to the table.
9 If you get a Status Code: 400; Error Code: ResourceNotFoundException error, check the region you specified in the Lambda implementation corresponds to the region of the DynamoDB table.

Creating the API Gateway
The API Gateway is used to create an HTTP endpoint which is the trigger for the Lambda function. Applications can communicate with this endpoint over HTTP.

1. Open de AWS console and navigate to API Gateway.
2. Create a new API. As the name use: LocationPersisterApi
3. Click Create API
4. Select Actions -> and click Create Resource, see screenshot_create_resource
5. As resource name use: devicelocation, and click Create Resource
6. Select Actions -> and click Create Method and select POST.
7. In the method details select as Integration Type: Lambda Function, Lambda Region (the region you created the Lambda Function in, and as Lambda Function: persistDeviceLocation (the name of the Lambda function))
8. Click Save and then OK
9. Click Test and paste a test message in the body. This can be the same message as used in the Test section of the Lambda function. After the body is filled-in, click Test. If everything is OK you should see a HTTP 200 status code.
10. Select Actions -> Deploy API and select [New Stage]. Specify prod as the stage name.
11. Click Deploy. Your API will be deployed so that it can be accessed from the outside world.
12. Navigate to the prod stage, expand the resources and select the POST method. Copy the URL after the 'Invoke URL' text in, for example, Postman. Execute an HTTP post with a test-message body. You should see a HTTP 200 status code (success).
13. If you now open the DynamoDB tables and list the items you should see several items added to the table.

Conclusion

AWS Lambda, DynamoDB and API Gateway is a powerful to provision functionality in the cloud without having to provision entire servers or more full-fledged managed services like elastic beanstalk. This post showed you how to use those AWS services to create A Lambda function which uses DynamoDB and make it available using API gateway.

Resources

- The full source code of the example project can be found on Github.
- AWS Lambda
- AWS DynamoDB
- AWS Api Gateway

IntelliJ TooltipRunner plugin

Live coding during presentations can be a powerful mechanism to captivate an audience and this is preferably done using a tool which is suited for this. This means an IDE which provides some sort of presentation mode to focus on the code at hand.

When watching Get a Taste of Lambdas and Get Addicted to Streams by Venkat Subramaniam Venkat uses this technique of live coding. What is especially useful is the that TextMate is setup so that the results of program execution are displayed as a tooltip.

Since I use IntelliJ instead of textMate I searched if there was something similar. Unfortunately there was not. There is the convenient presentation mode but not an option (that I can think of) to display the results of a Java main execution in a tooltip.

That is why I created a plugin which does the same for IntelliJ. The plugin can be found here or you can install it using the Plugin manager in the Preferences. You can find a demonstration of the plugin on YouTube.

The source code of this plugin can be found on Github: https://github.com/jcraane/intellij-tooltip-runner

Increase max open files in Elastic Search (OSX)

Elastic Search, Logstash and Kibana (ELK) is an end-to-end stack which provides realtime analytics for almost any type of structured or unstructured data. 

When importing large amounts of data using Logstash to Elastic Search (ES), the chances are that ES hit the limits of the maximum files it can open. This limit is seen as an error in the ES logs with the following description: (Too many open files)

To deal with this you can increase the maximum files ES (or any process) may open using the following steps:

1. First start ES with the following option: ./elasticsearch -Des.max-open-files. This wil show the maximum number of files ES is allowed to open, for example: [2016-02-22 06:44:09,558][INFO ][bootstrap                ] max_open_files [10240]

2. Now execute the following commands to increase the maximum number of files a process may open:

- sudo sysctl -w kern.maxfiles=32000 
- sudo sysctl -w kern.maxfilesperproc=32000

3. Execute the following commands to set the file limit for the terminal process (this is the terminal window to launch ES in)

- ulimit -Sn 32000
- ulimit -Hn 32000

For ES 1.7:

Start Elastic Search with the following command: ./elasticsearch -Des.max-open-files -XX:-MaxFDLimit=true

For ES 2.2

Execute the following commands:

export ES_JAVA_OPTS=-XX:-MaxFDLimit (this increases the maximum files the JVM is allowed to open by default, see JVM configuration for more information)

and then start ES with the following command: 

./elasticsearch -Des.max-open-files (max_open_files should be 32000 now)

Heap sizes

You may also need to increase the heap size of both Logstash and Elastic Search. 

To increase the heap of LogStash execute the following command before launching Logstash: export LS_HEAP_SIZE=2g

To increase the heap of Elastic Search execute the following command before launching ES: export ES_HEAP_SIZE=8g

When importing large files using Logstash, it may benefit to increase the number of workers to speed up the importing process. The default is 1. See the following example of the elasticsearch output plugin in Logstash:

elasticsearch {
     action => "index"
     hosts => ["localhost"]
     index => "logstash-%{+YYYY.MM.dd}"
     workers => 4
     flush_size => 1000
}

Android studio VS IntelliJ

Introduction

On May 16, 2013 at Google I/O, Google announced Android Studio to be the primary IDE for building Android apps which is build on the IntelliJ platform. Since both the community and ultimate edition of IntelliJ also support Android development, what is the difference between the two and how to choose? In this post I will focus on Android studio and the ultimate edition of IntelliJ, which requires a paid subscription.

I only focus on Android development. If you develop applications with a variety of different technologies, IntelliJ Ultimate edition is probably the best choice.

Added advantage

Let's make one thing clear: Android studio is an awesome IDE and for most of us it satisfies our Android development needs. But there are some features in IntelliJ which are lacking in Android studio which may come in handy when doing Android development. I will discuss those features below.

JSON support

IntelliJ Ultimate has full featured JSON support. If your Android application connects to a JSON endpoint, it may be worthwhile to have an editor with JSON support.

REST Client

IntelliJ Ultimate provides a REST client which can be used to execute URL requests to test available backend services.

Structural search and replace

Structural search and replace provides an advanced search & replace mechanism with knowledge about the source code. For example, I used this tool to rewrite the Google analytics V2 to Google analytics V4 in less then 10 minutes. Analytics code was scattered all over the place but structural search and replace lets you easily refactor such code.

Database support

If your application uses a relational SQLite database, a database editor is very helpful. IntelliJ Ultimate provides database support and also includes a connector to connect to an Android SQLite database which is really helpful to inspect the database of your application.

Advanced debugger

IntelliJ 14 provides an advanced debugger which shows variable values in the editor right next to their usages.

Analyze dependency matrix

The Analyze dependency matrix can be used to analyze the dependencies between projects and classes.

Conclusion

Although Android studio is a wonderful editor for Android development this post showed a couple of features in the IntelliJ Ultimate edition which may be worth the price when doing Android development.

Parse.com anonymous and registered users (Android)

Introduction

When writing a mobile application you mostly always need a way to store the information outside of the application itself so the data is accessible to not only the application itself on a specific device, but on every device the application is installed on and perhaps even web applications. This means the application will need some sort of a backend service/API to communicate with. This usually also means the application will use some sort of account/user management.

There are several so-called mobile backend as a service platform which facilitate this. The word mobile is a bit misplaced I think because the clients of those platforms are not always mobile devices.  I therefor call those platforms Backend As a Service (BAAS). Parse.com is one of those platforms.

This post talks about the way you can manage users with Parse.com and specifically how to use anonymous users and convert between an anonymous and registered user.

Anonymous user

A lot of apps require the user to register (create a user account) or login with a Facebook or similar account. If this is a required process, chances are that a certain group of users will not use your app because of the required login. If your app functionality allows, a possible work-around for this is to provide the app with anonymous user login. By using an anonymous user, the user of your app can experience all or most of the functionality of the app without requiring a user account. If the user likes your app, he/she can then sign-up for a registered account. Ideally, all of the data gathered during anonymous access, should be transferred to the registered account. Fortunately, the above functionality is fairly easy to implement using the Parse.com platform.

Enabling anonymous access

To enable your Parse.com app for using anonymous access, you have to do the following:

1. Enable anonymous access in the Parse.com console. Go to Settings -> Authentication and enable "Allow anonymous access".

2. Add the following code in the onCreate method of your Android Application class:

@Override
    public void onCreate() {
        super.onCreate();
        Parse.initialize(this, "APPLICATION_ID", "CLIENT_KEY");
        ParseUser.enableAutomaticUser();
}

By enabling automatic user, the call to ParseUser.getCurrentUser() always returns a user and thus is never null. You can check if the user is an anonymous user or a registered one by using the following code:

ParseAnonymousUtils.isLinked(ParseUser.getCurrentUser());

This can be useful to check if the sign-up button should be displayed or disabling some functionality which is only accessible to registered users.

Converting an anonymous user into a registered one

An anonymous user can be converted to a registered one. The data belonging to the anonymous user is also present on the registered one.

Before converting an anonymous user, there are some things to consider:

1. The username can not be left blank. You must explicitly specify a username and password on the user which is to be converted into a registered one.
2. It is adviced to save the anonymous user to the backend as soon as it is created. If this is not done and a call to saveInBackground is called on the registered user (after converting it from an anonymous one) a stack overflow is generated from the Android parse SDK. See also the following question on Stack-overflow (created by me): http://stackoverflow.com/questions/27595057/converting-an-anonymous-user-to-a-regular-user-and-saving

To save the user immediately after it is created, modify the Application code so that it looks like this:

@Override
    public void onCreate() {
        super.onCreate();
        Parse.initialize(this, "APPLICATION_ID", "CLIENT_KEY");
        ParseUser.enableAutomaticUser();
        ParseUser.getCurrentUser.saveInBackground();
}

The anonymous user can now be converted into a registered one with the following code:

findViewById(R.id.createUser).setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(final View v) {
                final String accountUsername = username.getText().toString();
                final String accountPassword = password.getText().toString();
                final ParseUser user = ParseUser.getCurrentUser();
                user.setUsername(accountUsername);
                user.setPassword(accountPassword);
                user.signUpInBackground(new SignUpCallback() {
                    @Override
                    public void done(final ParseException e) {
                        if (e != null) {
                            Toast.makeText(MainActivity.this, "Signup Fail", Toast.LENGTH_SHORT).show();
                            Log.e(TAG, "Signup fail", e);
                        } else {
                            Toast.makeText(MainActivity.this, "Signup success", Toast.LENGTH_SHORT).show();
                            final ParseUser user = ParseUser.getCurrentUser();
                            user.put("phone_no", "31612345678");
                            user.saveInBackground(new SaveCallback() {
                                @Override
                                public void done(final ParseException e) {
                                    if (e != null) {
                                        Toast.makeText(MainActivity.this, "Save data Fail", Toast.LENGTH_SHORT).show();
                                        Log.e(TAG, "Signup fail", e);
                                    } else {
                                        Toast.makeText(MainActivity.this, "Save data success", Toast.LENGTH_SHORT).show();
                                    }
                                }
                            });
                        }
                    }
                });
            }
        })

Please note that the data associated with the user in the saveInBackground call (after sign-up is successful) could also be associated immediately to the user before the signUp call. This saves an extra network call. The call to saveInBackground is pure for demonstration purposes.

Conclusion

This post showed the benefits of an anonymous user of a mobile app and how the anonymous user can be used with the Parse.com platform. It also showed code examples of how an anonymous user is converted into a registered one and the potential problems and solutions with it.

IntelliJ: the power of structural search and replace

Sometimes you run into a situation where you want to refactor some code but cannot use the regular refactorings. For example, take the following code:

jQuery("body").on("change", "#fontSelector", "change", function () {
    var selectedFont = jQuery("#fontSelector").val();
    layoutDesigner.selectFont(selectedFont);
});

I wanted to replace the above code with the following:

jQuery("#fontSelector").off("change");
jQuery("#fontSelector").on("change", function () {
    var selectedFont = jQuery("#fontSelector").val();
    layoutDesigner.selectFont(selectedFont);
});

Sure, I could rewrite this manually. But this takes a long time with dozens of such event handling constructs, all with different selectors, events and functions. Structural search and replace to the rescue.

But instead of writing how you could do this with structural search and replace, I recorded a little screencast which demonstrates the concept. The video can be found here: https://www.youtube.com/watch?v=Jb-YNgDClKg

Android: location based services

Introduction

Developing applications for mobile devices gives us a lot more opportunities for context based information than a traditional web application. One of the inputs for context sensitive information is the users current location. This post describes several ways an Android application can obtain the users current location.

Location API's

In previous versions of the Android SDK you had to manually implement a location service which abstracts away the underlying location providers (GPS or cellular based). This was not ideal since as a developer of an application, you probably are not concerned about the implementation details of obtaining a users location.

Fortunately Google's Location API's provide a much better way for working with location data. The Location API's provide the following functionality:

• Fused location provider which abstracts away the underlying location providers.
• Geofencing. Lets your application setup geographic boundaries around specific locations and then receive notifications when the user enters or leaves those areas.
• Activity recognition. Is the user walking or in a car.

Check for Google play services

Working with the Location API's require the presence of the Google Play Services application on the device. It is good practice to test for the presence of Google play services before using the API. This can be done with the following code:

protected boolean testPlayServices() {
        int checkGooglePlayServices = GooglePlayServicesUtil.isGooglePlayServicesAvailable(getActivity());
        if (checkGooglePlayServices != ConnectionResult.SUCCESS) {
            // google play services is missing!!!!
            /*
             * Returns status code indicating whether there was an error.
             * Can be one of following in ConnectionResult: SUCCESS, SERVICE_MISSING,
             * SERVICE_VERSION_UPDATE_REQUIRED, SERVICE_DISABLED, SERVICE_INVALID.
             */
            GooglePlayServicesUtil.getErrorDialog(checkGooglePlayServices, getActivity(), 1122).show();
            return false;
        }
        return true;
    }

Code listing 1

Code listing 1 shows how to check for the presence of Google play services. If the Google play services are not present a dialog is displayed giving the user the opportunity to download and install the Google play services application. This method returns false if the services are not found and can be placed around any code requiring Play services.

Obtaining the users location

The primary class for using the Location API's is the LocationClient. The first thing to do is instantiating the LocationClient and passing the required listeners. See the following code which is usually called from the onCreate from within an activity or onActivityCreated if the LocationClient is instantiated within a fragment.

locationClient = new LocationClient(getActivity(), this, this);

The parameters are:

1. The Context
2. ConnectionCallbacks. Defines the onConnected() and onDisconnected() methods.
3. OnConnectionFailedListener. Defines the onConnectionFailed() method.

When the LocationClient is instantiated, the next thing to do is calling the connect() method of the LocationClient. This is typically done in the onResume method. In the inPause method the disconnect() method is called of the LocationClient. This ensures the LocationClient is only active when the activity is running. Should you need constant tracking of the users location when the app is in the background, it is better to create a background service for this.

When the connect() method is successful, the onConnected() callback is called. In this method you can obtain the users last known location using the following method:

locationClient.getLasLocation();

Periodic location updates

Registering for periodic location updates involves slightly more work. The first thing to do is creating a new LocationRequest object. This object specifies the quality of service for receiving location updates. The following code demonstrates this:

private static LocationRequest createLocationRequest() {
        final LocationRequest locationRequest = new LocationRequest();
        locationRequest.setPriority(LocationRequest.PRIORITY_HIGH_ACCURACY);
        // The rate at which the application actively requests location updates.
        locationRequest.setInterval(60 * MILLISECONDS_IN_SECOND);
        // The fastest rate at which the application receives location updates, for example when another
        // application has requested a location update, this application also receives that event.
        locationRequest.setFastestInterval(10 * MILLISECONDS_IN_SECOND);
        return locationRequest;
    }

After the LocationRequest is created and the connect() method of the LocationClient is successful, the onConnected method is called. In this method the LocationClient is instructed to send periodic location updates to the application using the following code:

locationClient.requestLocationUpdates(locationRequest, this);

The parameters are:

• locationRequest. Specifies the quality of services of the location updates.
• LocationListener. Defines several callback methods including the onLocationChanged which is called when a new location is available.

Required dependencies

To use the Google play services in your application you have to define the correct dependencies in the build.gradle. There are two versions of the API: one for Android 2.3. and higher and one for Android 2.2.

Use the SDK manager to install the required packages. For Android 2.3 these are:

• Google play services
• Google repository

For Android 2.2 these are:

• Google play services for Froyo
• Google repository

So if your applications targets Android 2.2 you must use the Google play services for Froyo library. In your build.gradle specify the following dependency:

For Android 2.3:

dependencies {
    compile 'com.google.android.gms:play-services:4.0.30'
}

For Android 2.2:

dependencies {
    compile 'com.google.android.gms:play-services:3.2.65'
}

Testing with mock locations

To test with mock locations you have to do the following:

1. Enable mock locations in the developer options.
2. Download the sample LocationProvider example app: http://developer.android.com/training/location/location-testing.html
3. Modify the LocationUtils class with an array of locations you want to test with.
4. Install the LocationProvider sample app on your device.
5. Start the LocationProvider sample app.
6. Start the application you want to test the location functionality for.

A handy website to get the latitude and longitude of an address for testing purposes is: http://www.itouchmap.com/latlong.html

Conclusion

Working with location data in your mobile application can add a new dimension to the user experience. This article explains the steps needed to use the Google Location API's for obtaining the users current location.

MyBatis: mapping a map

As some of you will know I am a huge fan of MyBatis. I have used it in a lot of projects and it never failed me. I like how you are in control of the SQL and the flexibility this brings by mapping result sets to classes instead of mapping tables to classes.

Recently I wanted to map some columns from a table to actual typed properties of an object, and some columns to a property of type Map within that same object. Consider the following class:

public class Person {
    private String firstName;
    private String lastName;

    private Map dynamicProperties;

    // Getters and setters omitted.
}

and the following query:

select firstname, lastname, pref_1, pref_2, pref_3 from person;

The following resultmap maps the result of the query to the Person class:

<resultMap id="personDynamicProperties" type="map">
        <result column="pref_1" property="pref_1"/>
        <result column="pref_2" property="pref_2"/>
        <result column="pref_3" property="pref_3"/>
    </resultMap>

    <resultMap id="personResult" type="Person">
        <result column="firstname" property="firstName"/>
        <result column="lastname" property="lastName"/>
        <association property="dynamicProperties" resultMap="personDynamicProperties"/>
    </resultMap>

Java 7 try-with-resources

Java 7 provides better resource management for resources that need to be closed when finished working with, for example files, streams, database connection and sockets. This language construct is called the try-with-resources statement. The mechanism that makes this work is called the AutoCloseable interface. The Java 7 resource classes all implement this interface. The signature of this interface looks like this:

public interface AutoCloseable {
    void close() throws Exception;
}

It declares one method, close(), which is automatically invoked on objects managed by the try-with-resources statement.

Although Java 7 resource classes implement this interface, a lot of time the libraries you use do not because the library is not updated to use the AutoCloseable interface or the project cannot simply update to a newer version.

Most of the time this is easy to solve. Just subclass the resource that should be able to participate in the try-with-resources statement. Take the ITextRenderer (form the Flying Saucer project) as an example. When finished working with the ITextRenderer, the finishPDF() method should be called. Normally you would do that in a finally block. By creating a new class extending from ITextRenderer and implementing the AutoCloseable interface this class can participate in automatic resource management. The AutoCloseableITextRenderer looks like this:

public class AutoCloseableITextRenderer extends ITextRenderer implements AutoCloseable {
    @Override
    public void close() {
        super.finishPDF();
    }
}

Extending the original class makes the most sense since the subclass is an ITextRenderer. You would use composition if the class cannot be extended because it is final.

And this is how you would use it:

try (final AutoCloseableITextRenderer iTextRenderer = new AutoCloseableITextRenderer()) {
            ByteArrayOutputStream out; // contains the data to be converted to PDF, not shown here.

            iTextRenderer.setDocumentFromString(new String(out.toByteArray()));
            iTextRenderer.layout();
            iTextRenderer.createPDF(pdfOutputStream);
            pdfOutputStream.flush();
        }

Thats all. Please note that I did not throw an exception from the close() method in the AutoCloseableITextRenderer. The Javadoc of the AutoCloseable interface says the following about this:

While this interface method is declared to throw {@code Exception}, implementers are strongly encouraged to declare concrete implementations of the {@code close} method to throw more specific exceptions, or to throw no exception at all if the close operation cannot fail.

Height problem when rendering an ExtJs 4 application in a custom div

This post is verified with Ext Js version 4.

ExtJs applications can be run in the whole browser window or in a small part of a larger application, for example in a div. You may want to render an ExtJs application in a div if you have a common HTML structure with navigation capabilities outside the ExtJs application. To render an ExtJs application to a div do the following:

• create an HTML which contains the div where the application is rendered
• Create an app.js file which is the actual ExtJs application. 

Below is an example of an index.html with a div named appContent where the ExtJs application is rendered to:

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
    <title>Sample app</title>
    <script type="text/javascript" src="extjs/ext-debug.js"></script>
    <script type="text/javascript" src="extjs/ext-theme-neptune.js"></script>
    <script type="text/javascript" src="app.js"></script>
</head>
<body>
<div id="appContent" class="app-content">

</div>
</body>
</html>

And here is an example of app.js which renders the application to a div:

Ext.application({
    name: "SampleApp",
    launch: function () {
        Ext.create("Ext.panel.Panel", {
            renderTo: Ext.getElementById("appContent"),
            autoCreateViewPort: false,
            layout: {
                type: "hbox",
                align: "stretch"
            },
            id: "appContainer",
            listeners: {
                beforerender: function () {
                    Ext.getCmp("appContainer").setHeight(Ext.get("appContent").getHeight());
                    Ext.getCmp("appContainer").doLayout();

                    Ext.EventManager.onWindowResize(function () {
                        Ext.getCmp("appContainer").setHeight(Ext.get("appContent").getHeight());
                        Ext.getCmp("appContainer").doLayout();
                    });
                }
            },
            items: [
                {
                    xtype: "label",
                    text: "Sample App",
                    flex: 1
                }
            ]
        });
    }
});

Please notice the listeners section. What I noticed was that with some layouts (for example the hbox, vbox and borderlayout) the application did not occupy the whole size of the div. When I rendered the same application using a viewport the application size was a large as the browser window.

The beforerender listener fixes this issue. It basically sets the height of the application container to the height of the div and calls doLayout to update the view. This same logic is also added to the onWindowResize event to adjust the size of the application when the browser window is resized. This fixed the problem for me.

How to: web service mock with SoapUI

Introduction

A lot of organizations use web services (WSDL) for system integration. Often those services are developed alongside the application that uses those services. Instead of waiting for those services to be developed, the application which consumes the web service can develop a mock implementation of the service to test against.

Several strategies

There are several strategies for developing mock services from a WSDL. One of these strategies is using SoapUI. SoapUI is able to consume the WSDL and generate a mock service from it. After the mock service is created, the application in question can consume this mock service and start using it.

The advantage of mocking this way, is that the application makes a full HTTP network round-trip when calling the respective service. All relevant components within the application (WSDL client, parsing the response message) are covered.

Instead of describing how to mock a web service using SoapUI, I created a small screencast demonstrating this. The screencast can be found here: Web service mocks with SoapUI

I think this screencast is more clear and easier to follow than written text. To be effective I keep my screencasts shorter than 5 minutes. Descriptive subtitles are also included. Please let me know if you like this format.

Final thoughts

Besides speeding up development, mocks/stubs may also be used to put the system in a known state which enables automatic integration testing.

Reasons for IntelliJ

Introduction

I often get the question why I use Intellij in favor of another IDE, in this case Eclipse. Most of the time I answer that question by demonstrating some features of IntelliJ and showing how integrated everything is. This got me thinking about what are the actual reasons that I use it. This post will try to make that clear and help others decide if the switch is worth it or not.

Some background

I had been a long time Eclipse (7+ years) user before I made the jump to IntelliJ. Before Eclipse, I worked with Rational Application Developer, WSAD, JBuilder and Visual Age for Java. Compared to these IDE's, Eclipse was a joy to use. I could, for example, generate getters and setters, which was not possible in one of the older IDE's (we are talking about more than 10 years ago). Although I quite liked Eclipse I always thought there were some deficiencies. Mainly in the following areas:

• Why was there no core functionality bundled with the standalone Eclipse variant? For example Subversion and Maven integration.
• Why was it always painful to setup an Eclipse version to your liking with all the required plugins? With every new version I spent always nearly half a day setting up my IDE. This is unacceptable I think. The more plugins and functionality the harder it got.
• Updating to a new version was sometimes painful. Plugins that stopped working for example.
• I never quite liked the concept of a workspace. I already organize my projects on disk so I do not need a workspace concept.
• I did not like the idea of different perspectives. Why do I have to think about the context I am working in? For example: working with Java and Flex in one project. When I am in the Flex perspective my Java code completion/refactoring did work in Java files. Context should be file or even fragment driven.

Please note that the above are personal opinions and may vary between users. Despite of this I was quite productive in Eclipse and liked the performance of it. Also note that these observations are from a couple of versions back. Things may have changed.

Around 2007/2008, a colleague of mine introduced me to Intellij, I think it was version 7 back then. My first reaction was I don't need another IDE. He showed me some features, like code inspections, and I said I would give it a try. My main obstacle back then was the price. That year I also gave a talk at the Dutch Java User group conference. Every speaker received a free IntelliJ license from JetBrains. I then decided I would give it a try. After the first two or three days I thought I would give up. I had to learn all new key-bindings and I was less productive. I persisted and after a week or so I begun seeing the benefits of it. After version 7 I upgraded to 8, 9 without any problem. Things could be different. At the moment I work with the latest version, 12.1 EAP. Below are some of my reasons why I do most (if not all) of my development work in IntelliJ.

Major features

• It is an integrated solution. I do a lot of different development work with a lot of different technologies, for example: Java, HTML/CSS/JavaScript, Android, Grails/Groovy, Flex, Subversion, Git, Maven, Ant etc. This is all possible with IntelliJ out-of-the box. There is no need to install separate plugins, which saves me a huge amount of setup time. Just download and install it and you're good to go.
• The editor itself. I invest heavily in knowing all the shortcuts. By knowing all the shortcuts I can code very fast. The instant code completion (not having to hit Ctlr-space all the time) is a joy to work with. Just type a couple of characters and hit Tab to complete the code. When I generate code, the cursor almost always is in the correct position to begin typing again. No need to touch the mouse or whatever.
• Code inspections and analysis tooling build in. I find it important to keep my code clean and bug free. The build in inspections and the ability to auto solve them are a really nice addition. Besides this you also have a dependency matrix viewer to get a quick overview of the dependency structure of your application and a duplicate code checker.
• Live templates. Live templates greatly increase coding speed. To make the most of it, I highly recommend creating your own templates. This is very easy. Just select a piece of code and select Save as Live template from the Tools menu. Press Ctrl/Cmd+J to view the live templates.
• Maven/Gradle integration out of the box. Just import a Maven project and Intellij knows the modules, dependencies etc. You can easily generate a dependency diagram from the Maven pom file to view all the dependencies at a glance. See figure 1 for an example of the Maven dependency viewer.
• Some handy tools. I often use the database editor and the RESTful web service test utility. The database editor has code completion in SQL and table creation. With the RESTful web service tester you can easily test HTTP services. The response can then me immediately saved and formatted as JSON or XML.
• Powerful refactorings and structural search & replace. IntelliJ knows a lot about my code. For example in Android: when I rename an image in the values/hdpi folder, it also renames the corresponding images in the mdi and xhdpi folder but also updates my XML views and code references to that image.
• Tasks and Contexts. I use IntelliJ in combination with YouTrack (there are more issue trackers that IntelliJ can integrate with). It is really easy to start working on an issue. IntelliJ creates a new context that tracks the files that belongs to that specific issue. I can mark the issue in progress and when I commit my changes it takes the comments from the context and uses this as the commit comments. It also changes the status of the issue to resolved when done working on the issue. All from within the IDE itself, no need for context switching.

Figure 1: Maven dependency viewer

Smaller features

And then there are the smaller but just as important features which increase my productivity:

• Stacked clipboard. You can have multiple entries in your clipboard. Just hit Ctrl-Shift-V to show the clipboard stack.
• Column mode in the editor. This comes in handy when working with fixed structure files like CSV for example.
• Darcula theme. This is one of the best dark themes I encountered. A dark theme is especially useful when coding in the evening with the lights dimmed. It is less stressful for the eyes I think. See figure 2 for an example of the Darcula theme.
• Stack trace analyzer. Just copy a stack trace from the clipboard and IntelliJ analyses it and matches it with the code to easily navigate to the problem at hand.
• Unit test and coverage integration.
• And many more.

Figure 2: IntelliJ Darcula theme

Final thoughts

This article describes the reasons why I use IntelliJ as my primary development tool of choice. Please note that this is my personal opinion. Also, this is obviously not an exhausted list. I would like to hear from you why you choose IntelliJ.

Evaluating new frameworks, libraries and technologies

Introduction

New frameworks and libraries are developed so fast it is nearly impossible to evaluate all of them. This can make it hard to judge whether a particular technology is worth investing (time) in. This post gives you some tips which may help when evaluating new technology.

Conceptual level

Although there are many different libraries and frameworks, most of them can be grouped based on some conceptual differentiation. The following concepts may help to give an idea of this conceptual differentiation:

• Component based frameworks
• Request/response based frameworks
• Object Relational Mapping
• Map-reduce
• Key-value stores
• Messaging
• And many more... 

When evaluating a specific technology it helps to identify to which concept or concepts it belongs. Before further analysis of a particular technology, make sure you understand the concepts. Without this understanding it is almost impossible to do an objective analysis. Understanding these concepts requires an investment in time. Usually in the form of reading, watching presentations, experimenting, training and talking to others. Understanding the concepts will make it easier to objectively evaluate a particular technology.

Simpler is usually better

One of the key actions in evaluating a particular technology is to determine if and how it satisfies the given requirements. A lot of frameworks and libraries provide a 1 minute introduction, a 5 minute guide, a 30 minute tutorial and more. I really like these sorts of introductions since it usually gets you up-to-speed quickly. Besides getting up-to-speed quickly, a more important factor is the complexity of the given solution. I really like the idea of "Make things as simple as possible, but not simpler".

The starter guides, as mentioned above, present the readers with a relatively simple case or problem to demonstrate the key features of a given technology. To find out if a certain technology will fit I suggest to look at the complexity of the solution for the cases in the introduction guides.

In my opinion, if the solution for a relatively simple case or problem is complex and not easy to understand, how can a solution to a more complex problem be easy to understand? This complexity comes of course in various flavours:

• How much code do I need to write to get even the simplest things done?
• How many classes do I need to extend? 
• What and how much configuration should I write?
• What is the quality of the documentation?
• How can I easily test my solution?
• Etc.

If the solution to a simple problem is indeed simple and easy to understand, it is worth investigating more time to find out if the technology satisfies your requirements. If the solution to a simple problem is not simple and not easy to understand, think long and hard before investing more time in it.

Conclusion

New frameworks and libraries are developed so fast it is nearly impossible to evaluate all of them. To speed up the decision process make sure you understand the technology at the conceptual level. Once you understand the conceptual level, experiment with the technology using introduction guides if they are available.

If, from those introduction guides, solutions to a given problem are simple and easy to understand, it may be well worth spending more time investigating the particular technology for more complex problems. If not, proceed with care.

As always, your mileage may vary.

Sample generated melodies

Based on some questions, I uploaded a couple of sample melodies in MP3 format which are generated by my genetic algorithms melody generator. The generated sample melodies can be found here: http://code.google.com/p/melodycomposition/downloads/list.

The original article of the melody generator can be found here: Melody generator

Kinect within VMWare

Recently Microsoft announces the availability of the Kinect 1.6 SDK. One of the big enhancements in this release is the support for the Kinect device within a virtual machine (VMWare or Parallels)!

For a client we developed a Kinect application which was shown at the Dutch Floriade festival. More details about this project in a next post.

The problem with the old Kinect SDK was the inability to use the Kinect within a virtual machine. Because of this I had to create a bootcamp partition and install Windows on it just for the sole purpose of Kinect development. The 1.6 SDK solves this!

Below are the results of my tests with the Kinect within VMWare. I used the following system configuration to test on:

1. MacBook pro 2011 with 2Ghz core i7 and 8GB RAM..
2. OSX Mountain Lion.
3. VMWare Fusion 5 running Windows 7.
4. Visual Studio Express 2012.
5. Kinect 1.6 SDK.

First I tested the above setup with the Xbox Kinect. This did not work which I more or less expected since Mircosoft wants to push the Kinect for Windows for Kinect development.

Next I tested was the Kinect for Windows. After I connected the device I started the Shape Game sample application. The device was found and worked immediately! I also tested the Kinect with our own application and this also worked as expected.

Ont thing I noticed was that both applications did not run as smooth as when I ran the application under bootcamp. After examing my virtual machine settings I noticed I had given it only 1 cpu core (out of the available 8). This could be the problem. When I changed the maximum number of cpu cores the virtual machine may use to 4, things ran much smoother. There was no noticeable difference when running within a virtual machine or bootcamp. 

For more information about the 1.6 SDK see: http://msdn.microsoft.com/en-us/library/jj663803.aspx#SDK_1pt6_M2.

Conclusion
The new Kinect 1.6 SDK does work within a virtual machine if the Kinect for Windows is used.

IntelliJ SSR: replace constructors

Suppose you have a large codebase which uses a particular class named SomeClass (nice descriptive name :)). SomeClass has one constructor:

public SomeClass(String a, String b, String c, String d) {
    // Do something
}

This constructor is used at various places in the codebase.

At some point in time an update to SomeClass is made which introduced another constructor:

public SomeClass(String a, String b) {
    // Do something
}

Lets assume you want all calls to the 4 argument constructor to be replaced by the 2 argument constructor. The arguments of the 4 argument constructor should be used in the 2 argument constructor in the following way: new SomeClass(1, 2, 3, 4) becomes new SomeClass(3, 1).

See the following example:

public class UsageOfSomeClass {
    private SomeClass instance_1 = new SomeClass("a", "b", "c", "d");
    private SomeClass instance_2 = new SomeClass("d", "e", "f", "g");
    private SomeClass instance_3 = new SomeClass("h", "i", "j", "k");

    public static void main(String[] args) {
        SomeClass someClass = new SomeClass("1", "2", "3", "4");
    }
}

Should become:

public class UsageOfSomeClass {
    private SomeClass instance_1 = new SomeClass("c", "a");
    private SomeClass instance_2 = new SomeClass("f", "d");
    private SomeClass instance_3 = new SomeClass("j", "h");

    public static void main(String[] args) {
        SomeClass someClass = new SomeClass("3", "1");
    }
}

You can achive the following for an entire codebase with one single structural search and replace action. Use the following templates to achieve this:

Search template: new SomeClass($arg1$, $arg2$, $arg3$, $arg4$)
Replace tempalte: new SomeClass($arg3$, $arg1$)

Hit find and Replace All. Thats it!

IntelliJ: HTML and structural search and replace

I was asked to do some quick fix on some website. The website used a lot of table structures for laying out the page. To give an impression, the page had more than 50 tables each with 4 columns and 20+ rows.

There was no external CSS used. All styling of the tables was done with attributes or inline style elements. The styling was also not consistent throughout the page so each table looked slightly different.

Here is an example:

<table>
    <tr>
        <td width="200px">Some text</td>
        <td width="221px">More text</td>
        <td width="300px" height="100px">Example</td>
        <td style="width: 210px;">HelloWorld</td>
        <td style="width: 190px;">More text</td>
        <td id="test" style="width: 222px;"></td>
        <td id="test2" class="right" style="width: 200px;">Java</td>
    </tr>
</table>

Without rewriting the whole site at once I wanted to remove all the inline styles so I could use css to style all the tables the same. That at least looked more attractive in the short run while in the long run we could focus on a complete redesign of the site.

The first thing I wanted to do was replace all the td elements with inline styling with plain, empty td elements. As written in my previous posts, IntelliJ's structural search and replace features fits this use case nicely.

In the structural search and replace dialog I used the following:

Search template:

<td $width$="$value$" $style$="$value$">$text$</td>

Replace template:

<td>$text$</td>

In the Edit variables dialog I specified a minimum count of zero for the $width$, $style$ and $text$ variable. The $value$ variable has a minimum count of one. For all variables I used a maximum count of one.

The template searches for all td elements with zero or one width or style attributes (or both) and with or without text. The replace template replaces the td found with the search template in a plain td. The output of the HTML after the search and replace is run is:

<table>
    <tr>
        <td>Some text</td>
        <td>More text</td>
        <td>Example</td>
        <td>HelloWorld</td>
        <td>More text</td>
        <td></td>
        <td>Java</td>
    </tr>
</table>

After this replacement I could just insert the appropriate CSS and style all tables consistently. Without structural search and replace I think such scenario's take considerably longer to execute.

Hope you enjoyed it.

If you have other interesting structural search and replace use cases please share them.