The (accidental) political software developer

As a software developer, the chance to discuss politics is high at the coffee machine or after a couple of beers in the evening but not while writing code. Somehow the last few weeks proved me wrong, I managed to discuss controversial borders and disputed countries in more than one occasion. And all thanks to the new ubiquitous geolocation and image recognition services.

The Hong Kong user

The founders of RaceBase World, a service to discover and rate running events around the world, are based in Hong Kong. And they were not too impressed when their profile page stated as home country China. The geolocation labeling was provided by Mapbox, one of the largest provider of custom on-line maps. While the labeling might be justified – most of their users consider Hong Kong to be part of China – the choice was controversial for many runners based in the territory. And using the official name, the Hong Kong Special Administrative Region of the People’s Republic of China, is not really a feasible option.

Shenzhen anyone?

And it’s not the only service affected.

When I then uploaded a trail picture taken in Hong Kong but not far from mainland China on OneMediaHub (a cloud solution provided by Funambol) the result was more bizarre. The picture was labeled with the location “Shenzhen, China”. Even if downtown Shenzhen is not exactly a paradise for trail running and it is quite far away.

imageedit_4_5469168402In this scenario the problem was in the algorithm used to match EXIF data and the accuracy of the open source geolocation database used, GeoNames.

In the same way, a picture taken in the West Bank, not far from Jerusalem, has on the location “Jerusalem, Israel”. Again, the author was not too happy.


Is that really so bad?

Most of the geolocation services are pretty accurate and the error margin is very low. The vast majority of the users are hardly affected by the issues above, something we call corner cases. And even if one of your summer picture get tagged with the next town on the Costa Brava you are hardly going to complain. Or be offended. You might not even notice the bug.

But the problem is that a significant percentage of those scenarios where the algorithm fails or where there is a controversial decoding is in disputed territory or partially recognized states. And that introduces some challenges for the developer who does not want to deal with politics while writing code.

It’s only geolocation!

Actually even a simple signup form where the user has to choose the country might be controversial. Not everyone in the world sadly agrees on the status of Kosovo. Or Palestine. Or even their names.

Screenshot from 2017-04-16 13-43-36

Google uses “Palestine” (but label the field location) while Amazon goes for a neutral “Palestinian territories”.

Screenshot from 2017-04-16 13-47-21

Relying on the official UN status might be a safer option, but it does not make local users (or your web designer) very happy either. Let’s go back the Mapbox example with RaceBase World.

Screenshot from 2017-04-15 21-36-40

Mapbox works for the Palestine Marathon and make most (if not all) the runners attending the event happy but let’s assume a marathon is taking place in Simferopol, the largest city in the Crimean peninsula. Would most locals be OK with Ukraine as the country? Runners in Germany and runners in Russia have usually a different option about the status of Crimea. And there are many similar examples without even considering war zones.

Screenshot from 2017-04-16 12-51-40

How to fix those issues?

As a developer, if you have only a local audience it’s relatively easy. And you can minimize the controversies. If not, you can have some workarounds or hacks for challenging names or simply hide them (pretend that automatic decoding did not work or just show the city name). Racebase World for example now shows Hong Kong for new registrations in the autonomous territory.

Better, but with a significantly higher development costs, you could show localized names according to where the audience is. Or rely on localization to mitigate the issue (different names in different languages)

But at the end of the day the big players drive the geolocation databases and they care more about where most of their users are. When “2.8 million people took part in marathons in China in 2016, almost twice the number from the previous year”, as the Telegraph recently reported, it’s hard to argue with Mapbox’s approach on what China is and what China is not. Runners in Hong Kong might not be their first audience or growing market.

How can I test my application?

If you want to test how your website performs in critical area, you do not even need real pictures, just edit the EXIF data of a random picture using Photo Exif Editor or similar applications and enjoy the challenge. And you are read to go.

How does it work with AWS services?

What about Amazon and AWS services? Any way to limit or keep the above issues under control? This will be covered soon in the second part of this post.

Build a Alexa skill in (about) one hour

Since a few weeks ago the Amazon Echo Dot is finally available in Germany, the first not English speaking market. Time for me to test the cloud-based virtual assistant and see if it is really possible to build a trivia skill in under an hour as Amazon claims. And hopefully find out a bit more of how AWS Lambda is used by Alexa.

Any other benefit?

A new challenge aside, Amazon offered two main benefits to develop news (German) skills for Alexa.

  • developers who published a German Alexa skill in March could get an Alexa hoodieScreenshot from 2017-04-03 17-20-19

What’s the challenge?

No real technical challenge. Setting up a few trivia sentences in German was the hardest task. Never wrote a Node.js Lambda function before (I am more familiar with Java and Python) but there was no really any code to be written, it was simply a matter to change the template that Amazon set up. The goal was simply to get a meaningful skill live in the shortest time frame possible.

How long did it really take?

About one hour to set up the skill. A little longer to decide the topic, find an easy name (lauf lauf) of the trivia and the keywords to trigger the skill. And less than two hours to collect a few running trivia and write them in German. And submit the new skill for certification. All together a not too long evening in front of the laptop. The skill was approved after less than four days.


How does it sound?

What about privacy concerns?

You might not be a strong supporter of virtual assistants always in stand by mode. I do not have Alexa on all the time, I do not need to. I just got  a timer to switch it off at scheduled times and I changed a bit the default configuration  to avoid storing all the information on the Alexa application too. Note as well that you do not need to have a real Echo device to develop a skill but testing it in a real scenario is the fun part.

Was it worth it?

Yes. Not a technical challenge (it’s really too easy to set up a trivia) but a useful and interesting introduction to the world of Alexa and Lambda integration. And still impressed that in one evening I got something I was not familiar with from (almost) zero to live. Next step is to build a real skill, possibly using the Alexa Skills Kit SDK for Node.js

Base performance and EC2 T2 instances

Almost three years ago AWS launched the now very popular T2 instances, EC2 servers with burstable performance. As Jeff Barr wrote in 2014:

Even though the speedometer in my car maxes out at 150 MPH, I rarely drive at that speed (and the top end may be more optimistic than realistic), but it is certainly nice to have the option to do so when the time and the circumstances are right. Most of the time I am using just a fraction of the power that is available to me. Many interesting compute workloads follow a similar pattern, with modest demands for continuous compute power and occasional needs for a lot more.

It took a while for the users to fully understand the benefits of the new class and how to compute and monitor CPU credits but the choice between different instance types was very straightforward.

A bit of history…

Originally there were only 3 instance types (t2.micro, t2.small and t2.medium) and base performance, RAM and CPU Credits were very clear, growing linearly.

Instance     Base    RAM    Credits/hr

t2.micro     10%     1.0     6     
t2.small     20%     2.0     12     
t2.medium    40%     4.0     24

And price too. A t2.medium was effectively equivalent to 2 small instances or 4 micro instances. Both as credits, base rate and price. So far so good.

At the end of 2015, AWS introduced an even smaller instance, the t2.nano but the approach was still the same:

Instance     Base    RAM    Credits/hr

t2.nano      5%     0.5     3

Still same approach, nothing different.

Now large and even bigger!

But AWS extended the T2 class in the large range too, having first a t2.large in June 2015 and t2.xlarge and t2.2xlarge at the end of 2016. A lot more flexibility and a class that can cover many use cases with the chance of vertical scaling but finally the linear grow was broken:

Instance  RAM    Credits/hr Price/hr

t2.large        2        8      $0.094
t2.xlarge       4       16      $0.188
t2.2xlarge      8       32      $0.376

So far so good, the price per hour doubles according to vCPU and Memory. So a t2.2xlarge is equivalent to 4 t2.large. But what about the base performance?

Instance    Base Performance

t2.large      60% (of 200%)
t2.xlarge     90% (of 400%)
t2.2xlarge   135% (of 800%)

A t2.2xlarge is not equivalent to 4 t2.large.

I have a better base rate that I can run forever as average for cCPU running 4 nodes of t2.large (I would be able to average 30% on every vCPU) versus running a single t2.2xlarge (where I have a base performance of less than 17% on every vCPU) for the very same price.

The bigger you go, the lower the base performance by vCPU is.

So what?

Even without the loss in term of base performance, you have many reasons to choose more small instances: better HA in multi AZ, easier horizontal scaling, better N+1 metrics

But with T2 large+ instances even the AWS pricing strategy pushes you away from a single instance.

Unless you have an application that definitely benefit from a single larger t2 instance (for example a database server), spread out your load across smaller instances, with the T2 class you have one more reason for that.

The recently announced instance size flexibility for EC2 reserved instances makes it even easier to adapt the instance type even if you have a lot of RI capacity.

A wish list for the Amazon Elastic Transcoder

1 – Real-time video encoding

There is no real-time video encoding with the Amazon Elastic Transcoder.

How long does it take to transcode a job? It depends, usually not too long but if you need real-time or almost real-time encoding you need to look somewhere else or wait and hope they will add the feature in the future.

Over a year ago Amazon paid apparently $296 million  to acquire Elemental Technologies, a company that provides real-time video and audio encoding and still operates as a standalone company. But who knows, maybe there are some hopes for real-time video encoding in the AWS offer in the future.

2 – Filter out items that already match the preset

Let’s start with a simple example to clarify the problem. Let’s say I submit a job for a video (for example, Resolution 1920 x 1080, Frame Rate 29.97 fps, File Size, 27.4 MB, Duration 0:00:13.980) and I use a simple preset like “Generic 480p 4:3” (1351620000001-000030) My output is the expected:

Output Key playback.mp4
Output Duration 0:00:14.095
Output Resolution 640 x 360
Frame Rate 29.97 fps
File Size 1.7 MB

I pay one minute of encoding, I have a nice output and I am a happy customer. I now take the output and reiterate the process. Same pipeline, same preset, just a new job. I might hope to have the output identical to the input – the Amazon Elastic Transcoder has transcoded it, from the metadata of the output it’s obvious. Instead a new output is generated, I pay one more minute and I can keep iterating. The output every time is similar but the quality decreases.

OK, I submitted the job but I am not sure it makes sense to generate an output that is not as good as the input and that already matches the preset as best as your FFmpeg transcoder could. Would not be better to avoid any transcoding operation when the input is already in a “margin of error” of the specs of the preset? Output files cannot (obviously) be better than input ones. But AWS will try and charge for it anyway.

All this might sound like a corner case, but let’s think of a scenario where your end users upload the videos from their iPhone or tablets or laptop. And you are looking for surprises if you do not have good filtering in front of the pipeline.

3 – Processing a few seconds of video

Actually that is not true, the Amazon Elastic Transcoder can process video of any length, even just a few seconds. Whatever they are 6.5 second looping videos (Vines) or short video from Snapchat or whatever other source at a higher resolution. But it might not be the most effective way and it can be quite costly.

“Fractional minutes are rounded up. For example, if your output duration is less than a minute, you are charged for one minute.”

So for your 3 seconds you pay 60 seconds for every output. Any way around it? There are two more efficients way to process short videos. You can use the new Clip Stitching feature that allows you to stitch together parts, or clips, from multiple input files to create a single output and fractional minutes are then better rounded up. But you then need to handle the splitting of the output yourself. Or you can replace the Amazon Elastic Transcoder with aws-lambda-ffmpeg or similar open source projects based on AWS Lambda + FFmpeg and that are more cost effective for very short videos. You can then determine the minimum length that triggers the job on the Elastic Transcoder.

4 – Run a spot job & bidding price

I can bid on the price of a EC2 instance. I can run Elasticsearch on Spot Instances. I can rely on cheaper spot options on a Amazon EMR cluster. But I have no real chance to lower the costs of running Amazon Elastic Transcoder. I cannot (at least automatically from the console) commit to a certain amount of encoded hours to have a reserved capacity at a lower price. Many times, when I do not need real-time video encoding, I do not have a strong constraint of having the items in just a few minutes. Amazon Elastic Transcoder does a great job to process them (usually) in a few minutes but I would mind the option to bid a price or have a significantly longer queue time in exchange of a lower price when the service is less used. To achieve that, you are back to a cluster of FFmpeg running on EC2 instances.


5 – Encoding available in every regions

The Amazon Elastic Transcoder is currently available only in a subset of regions (8) and in Europe only in Ireland. From a technical point of view, you can always configure the Amazon Elastic Transcoder to handle the S3 bucket in one region and run the encoding in a different one. Latency is not that big of an issue, given that it’s anyway an asynchronous operation. And the cost of data transfer is as well negligible compared to the cost of the service itself. But the blocker is often a legal requirement and you cannot process a user video outside a specific region where the data is stored.

Enabling IPv6 on existing VPCs and subnets

AWS announced in December 2016 IPv6 Support for EC2 Instances in Virtual Private Clouds  in the US East (Ohio) Region only. At the end of January they finally extended the support in every AWS region, as for the post AWS IPv6 Update – Global Support Spanning 15 Regions & Multiple AWS Services.


How do you benefit from the new feature? Before creating IPv6 Application Load Balancers or EC2 instances in your VPCs, you need enable IPv6 support for the VPC and the subnet(s). Yes, you do not need to recreate the subnets. In case you have many VPCs to enable, it’s easier to rely on the AWS Command Line Interface.

Enable IPv6 using the CLI

Let’s say you have a VPC with subnets called staging, if you accept the default AWS range and distribute the subnets in a simple way, you just need a few lines in bash to enabled IPv6 for the VPC and all the associated subnets


vpcid=$(aws ec2 describe-vpcs --filters Name=tag-value,Values=$vpc_name 
| jq .Vpcs[].VpcId |  sed 's/"//g')

echo "Enabling IPv6 for VPC $vpcid"

aws ec2 associate-vpc-cidr-block --amazon-provided-ipv6-cidr-block --vpc-id $vpcid

ipv6range=$(aws ec2 describe-vpcs --filters Name=tag-value,Values=$vpc_name | 
jq .Vpcs[].Ipv6CidrBlockAssociationSet[].Ipv6CidrBlock | sed 's/"//g')


echo "IPv6 VPC range is $ipv6range"

subnets=$(aws ec2 describe-subnets--filters Name=vpc-id,Values=$vpcid 
| jq .Subnets[].State | wc -l)
while [  $COUNTER -lt $subnets ]; do
     subnetid=$(aws ec2 describe-subnets --filters Name=tag-value,Values=$vpc_name* 
| jq .Subnets[$COUNTER].SubnetId | sed 's/"//g')
     echo "IPv6 subnet $subnetid range $ipv6rangeprefix"
     aws ec2 associate-subnet-cidr-block --subnet-id $subnetid --ipv6-cidr-block $ipv6rangeprefix

You can perform manually all the steps above on the AWS console but as usual it is easier to handle multiple AWS accounts or deployments using the AWS Command Line Interface. You can as well loop and update all your VPCs in a single script.

AWS Professional level recertification

Today I cleared the recertification exam as AWS Certified Solutions Architect – Professional. And I am very happy. I passed my first certification with AWS early in 2013 and I have been through a few AWS exams since then as all certified architects are required to renew their certifications every two years.
Taking a recertification exam certainly help in keeping up to date and force any cloud specialist to review technologies he does not use every day and widen his knowledge of AWS patterns and best practices. And that’s incredibly valuable. But according the Get Recertified page, the goal the recertification process is something else:

AWS releases a growing number of new features and services each year. To maintain your AWS Certified status, we require you to periodically demonstrate your continued expertise on the platform though a process called recertification. Recertification helps strengthen the overall value of your AWS certification and shows customers and employers that your credential covers the latest AWS knowledge, skills, and best practices.

Unfortunately this is not the case yet. The AWS Certified Solutions Architect Professional Exam Blueprint has not significantly changed since the first time the exam was introduced. No AWS Lamdba, no Amazon API gateway, no containers. The focus is still on AWS Pipeline or Amazon Simple Workflow more than serverless technologies. And a few questions still discuss the benefits of Amazon S3 Reduced Redundancy Storage (RRS), a storage class that is not so valuable anymore.