AWS SAM local start-api, connecting to db/service when both are running in Docker Desktop

I am current building/testing a node/NestJS based API, that I am deploying to AWS Lambda.
I chose AWS SAM for the deployment tooling, as the integration is simple, lightweight and declarative.

Running the service locally, npm run start:dev, works perfectly and connects to the DB, mongo, hosted in Docker without a problem

Before deploying, I webpack the project to 1 file and run it in a lambda equivalent environment, which also runs on Docker, using "sam local start-api" and here is where the problem appears... 

My service is unable to connect to mongo, when both are running in Docker.

After the usual questions... is MongoDB running, can I connect to it, etc, I knew there was an issue.

Initially all I got was:

Function 'xxxxxx' timed out after 3 seconds

Really helpful.

After some messing around, I eventually reset the SAM timeout, retried and got:

MongoServerSelectionError: connect EHOSTUNREACH 127.0.0.1:27017

Now we are getting somewhere... it is unable to reach mongo.

To cut a long story short, after trying many different things, I eventually found the correct IP to call Mongo running in Docker. It's right down the bottom of the config if you are interested.

More importantly, I then looked up how to extract it directly...

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mongodb

Call that, stick the IP address in your environment variables, and tada.. bob's your aunties brother.

Should also work for any other db / service that you need to connect to, when running it all in Docker.

SE-Linux: check and deactivate

I quite often run into security problems when testing stuff on my machine.
This is to remind me how to deactivate SE-Linux, so I can see if it is causing the problem...

sudo getenforce
> Enforcing

If "sudo getenforce" returns "Enforcing", you have SE-Linux, so turn it off.

sudo setenforce 0

then

sudo getenforce
> Permissive

... now test the thing...

making Makefile Self documenting

I have use 'make' on the last few projects, just to control all the CMD line build tools we now need to use.

I found a neat trick a little while ago, that automatically shows you what CMDs are available.

#
# Why use Makefile?
# because you get help lists & auto-complete on complex commands
#

help:           ## Show this help.
 @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'

# linebreak
: ## ======================================================================

# make all output silent - ie: no CMDs shown
#.SILENT:

gitStatus: ## show GIT status
  @git status -b --column -s

Simple copy this into the top of your Makefile, and add comments as shown using ## after the cmd name.
Now when you call 'make' without any arguments, you will see a list of available commands with their descriptions.

A brilliant tool for projects with loads of cmd line tools to remember.

How to get AWS EC2 instance metadata from Java

I have recently found that I can access EC2 machine instance metadata using curl.
Which is brilliantly useful.

However, I wanted to get the same data inside my Java applications.
So I build a utility to make it available...

package com.mendeley.weblet.oauth.utility;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.client.apache4.config.DefaultApacheHttpClient4Config;

import java.net.URI;


/**
 * Utility class to access EC2 instance Meta-data
 *
 * @See (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
 */
public class EC2Metadata {

    /**
     * Some of the Metadata types
     */
    public enum Type{
        ami_id("ami-id"),
        ami_launch_index("ami-launch-index"),
        ami_manifest_path("ami-manifest-path"),
        block_device_mapping("block-device-mapping/"),
        hostname("hostname"),
        instance_action("instance-action"),
        instance_id("instance-id"),
        instance_type("instance-type"),
        kernel_id("kernel-id"),
        local_hostname("local-hostname"),
        local_ipv4("local-ipv4"),
        mac("mac"),
        network("network/"),
        placement("placement/"),
        public_hostname("public-hostname"),
        public_ipv4("public-ipv4"),
        public_keys("public-keys/"),
        reservation_id("reservation-id"),
        security_groups("security-groups"),
        services("services/");

        private String name;

        private Type(String name){
            this.name = name;
        }
    }


    /**
     * Get metadata using Type enum
     *
     * @param type
     * @param timeout
     * @param defaultValue
     * @return
     */
    public static String retrieveMetadata(Type type, int timeout,String defaultValue) {
        return retrieveMetadata(type.toString(),timeout,defaultValue);
    }

    /**
     * Get metadata by String value
     * Allows further metadata to be retrieved.
     * See AWS documentation for more info.
     *
     * @param type
     * @param timeout
     * @param defaultValue
     * @return
     */
    public static String retrieveMetadata(String type, int timeout,String defaultValue) {
        try{
            URI uri = URI.create("http://169.254.169.254/latest/meta-data/" + type);
            System.out.println(uri.toString());

            ClientConfig config = new DefaultApacheHttpClient4Config();
            config.getProperties().put(ClientConfig.PROPERTY_CONNECT_TIMEOUT,timeout);

            Client client = Client.create(config);
            WebResource webResource = client.resource(uri);

            ClientResponse response = webResource.get(ClientResponse.class);

            String results = response.getEntity(String.class);
            return results;

        }catch(Throwable t){
            return defaultValue;
        }
    }


    public static void main(String[] args) {
        String myEC2Id = retrieveMetadata(Type.instance_id,1000, "null");
        System.out.println("The Instance Id is " + myEC2Id + " .");
    }

}

I will try to get it into Github at some point, until then, please use it as you see fit.
Obviously I take no responsibility for the end of the world as you know it, should it occur.

Using ENYOjs to build a Chrome App, a few pointers

As per usual, I am trying something new... discovered a few problems, then solutions, and need to remind myself about the solutions for the next time.

I am building a Chrome App using ENYO, partly because I actually need an app right now, and partly because I think ENYO will make a perfect quick build solution for the next time.

Anyway...

Problem 1: document.write error
Solution: use renderInto.

var app = new UMLEditor({name: "app"});

app.renderInto(document.body);        

Problem 2: localstorage warning
Solution: refactor to use chrome.local.storage
   Open: source/data/sources/localStorage.js
   Change: e.localstorage to chrome.local.storage


Simple problems, simple solutions, but annoying to look up again next time.

Groovy Eclipse compiler versions and ShortTypeHandling

Just a short note to remind myself.

If you come across a classNotFoundException: org.codehaus.groovy.runtime.typehandling.ShortTypeHandling

Then the chances are you have 2 versions of Groovy being used.
1 pre Groovy 2.3.5, and 1 after it.

Reminder: use this to investigate (remember to look at parents too)
mvn dependency:tree -Dverbose

I believe this is a Java 8 compatibility change, but would need to investigate to be sure.

Anyway, go here to find out more:
http://glaforge.appspot.com/article/groovy-2-3-5-out-with-upward-compatibility 

UML Diagram Editor is ready

I have been working on a small chrome App, a tool for building UML diagrams from text. It leverages PlantUML, which I have found to be a great tool, but seriously lacks a decent desktop app to use it. So I build one, or at least the start of one.

You can find it here:
UML Diagram Editor

I have been teaching my colleagues to create Sequence Diagrams to plan agile stories. They can capture, communicate and confirm all the tasks required for a story with all the participants and interested parties, before even creating the tickets to achieve it.

As far as clear communication goes, I now see sequence diagrams as equal importance to using BDD style acceptance criteria. As both allow everyone up and down the technical/business chains to understand exactly what it planned, and what is required.

To use sequence diagrams for you project...

Keep it simple, only one feature per story.
It is more likely to finish on time, and easier to describe fully.
Any scope creep goes into a new story, to be prioritized into the backlog.

When planning the story, simply create a sequence diagram describing the flow that will occur when the feature is finished. Include UIs, acceptance criteria, validation, message formats, button clicks, and anything else you believe is needed to describe the solution you are going to build.

Now just look at your completed diagram, the tasks should fall out of it very readily, separated by component, and position in the flow. Dependencies will also be very obvious, so those tasks can be done in order.


I have found this to be a superb solution to the issue of communication within a team, as well as externally, as it provides an easy point of focus and discussion. When it is easy to edit or modify the UML diagram, it becomes a live document, and the only one needed if the tasks are kept small.

Why you want to build your own platform. PAAS on top of IAAS.

The problem:

PAAS and auto-scaling are brilliant, I love them, but I hate the hidden price...Vendor lock-in.
IAAS and cloud VMs are great, I love them, but I hate the hidden price...Vendor lock-in.

Neither solution works in the long run.

Either we run into a missing facility in the PAAS, or our costs scale out of control as IAAS gains traction with the developers, testers and everyone starts spinning up VMs all over the place.

We end up creating a Heath Robinson-esque machine, solving each little problem by adding one more part to our monster.  The complexity grows and grows and grows. Soon enough no more features can be added as keeping the machine running takes all our time.

Eventually we are going to realise that we need more/better/bigger and will have to change everything or go and find it somewhere else and we still have to change everything.

So vendor lock-in is the biggest elephant in the room, and we face it where ever we look at the moment. Yes, you can have all their shiny features, but you must create everything their way, and woe betide you if you think you can change provider easily.

Everything we do becomes tightly coupled to the providers infrastructure or systems:
development, deployment, testing, roll-back, DevOps, Sysops, metrics, and the list goes on...

There are ways around these issues, but they always feel like edge case coding, and I know another edge case is waiting just around the corner, in fact I have 3 in the backlog.

Are you ready? Here are the words, just in case:
"We welcome our Heath Robinson overlords..."

The dream:

For a long time, I have had a dream ... of a system that allows everyone to get the best of all worlds, without any group suffering to support other, and everyone is capable of working to improve their conditions.

• my applications should be able to live anywhere
• without requiring a complete refit, refactor and rebuild just to move home
• I can code it, test it, build it, deploy it and manage it easily, uncoupled from the infrastructure
• No more herculean efforts just to get tests to run locally or in the build
• Use the best of breed as standards, but avoiding the Heath Robinson effect
• A simple configuration system that has defaults for everything, yet all can be overridden
• Simple to change, and old parts replaced or new injected without downtime

How to do it?
This thought has bothered me for a long time.

The solution:

Eventually, I fell back on time honoured solution.
To de-couple 2 things, create an abstraction layer between them.

An Infrastructure Abstraction Platform: IAP.

Create an  abstraction layer, a platform for my code, that runs on top of any IAAS, that can manage itself, and all the applications inside it, as well as interface with the containing IAAS, but avoid coupling my applications to the IAAS. Everything external to my platform and applications should be attached through discovery and configuration, so an IAAS provider change should only require a redeploy and a config change.

If you have any idea how much time/effort this would take, you can understand my reluctance to even think about starting it. That was until I discovered Vertx.

Next:

[Coming soon: What Vertx lets me achieve]
[Coming soon: Why I use ENYO to build UIs]
[Coming soon: Why I can now build my own platform on IAAS]

Edit Makefile in Intellij

My projects now tend to use a lot of different tooling.
eg: Vagrant, docker, Node, Bower, CURL, ANT, Maven, Gradle and a bunch of other custom tools that I have to remember how to use. Worse still we are moving to mass production, so I will need to remember them across projects.

To solve this, I have recently started using Makefiles to aggregate all the command line stuff. Now I can store the rarely used commands and port them from project to project without needing to go and look them up again.

Anyway, I need to edit a Makefile in Intellij IDEA.
Easy right...

Nope, I now keep getting:
Makefile:17: *** missing separator.  Stop.

FYI: The Cause of this error: MAKE requires a TAB at the start of each command line, but Intellij has converted all TABs to multiple Spaces. You can change the Makefile to use space as indents, but it looks horrible.

Anyway, I found a few solutions, and this is mainly to remind myself. But if you happen to find this post and it helps, please let me know, or worse if it fails, do please let me know in the comments, as I will look into it.

Firstly, install the C/C++ plugin from the plugin repositories.
This adds Makefile syntax checking and highlighting.

Second: to sort out the tab converting syntax problem, open your Makefile and then...
EDIT -> Convert Indents -> To Tabs

FYI: to see the Spaces/Tabs, I used "Show Whitepaces", by...
HELP -> Find Action -> "Whitepace"
which shows the characters in the active editor.

Easy Design-By-Contract for Java and Javascript

Why use Design by Contract?

I like a very cut down form of Design By Contract. I use it to validate the arguments passed to a function. So if an expectation suddenly fails, my code will immediately drop me into the debugger if I am developing, or throw an Error with an actual meaningful message in Production.

I have been frustrated for a while by the number of lines of code (complexity) needed to implement Design By Contract in either Java or JavaScript, or the need to use recompiling frameworks so the output code is modified.

This frustration has finally come of age, and with some help from James Gee, we have almost finished working on ExceptionExtensions for both Java and JavaScript.

There are 2 advantages to using this:

1. It is designed to read like natural language, so code is much easier to read and write. Parameter validation in java REST interfaces, or function expectations in a JavaScript event handler, both become easy to write, and more importantly 6 months later, I can read it as easily as English, or my colleague can read it tomorrow without needing to ask me WTF?
2. Debugging an contract expectation failure become simplicity itself. Isolating and identifying the causes of bugs is so much easier if your code simply stops at the point of failure and makes a fully scoped stack available, without the need for complex conditional breakpoints.

I know it sounds simple, but using ExceptionExtensions and Design by Contract has massively improved the readability of my code and revolutionized my ability to isolate and identify the causes of a bugs.

Java:

import static com.techmale.exception_extension.ExceptionExtensions;

public void myMethodName(String arg1, int arg2){
    IllegalArgumentException.when(arg1 == null,"arg1 is NULL");
}

To debug: Simply set a break point in the ExceptionExtensions src code and run your code in debug mode. You will drop into step debugging without needing to set up any complicated conditional debug points.

Javascript:

function myFunc(arg1,arg2,arg3){
    arg1 = arg1 || "defaultValue";
    Exception.when(!!arg2,"Arg2 is mandatory");
    Exception.when(arg2 typeof != "Number","Arg2 must be a Number");
    Exception.when(arg3 != 100,"Arg3 must be a 100, is actually %s", arg3);
    
    // .. the rest goes here
}

To debug: Just make sure your console is open, as debugging is automatic out-of-the-box. If you want to turn it off for a production system, just read the instructions on GitHub, or the post below on conditional evaluation.

The Conclusion?
The price: A really simple change to your coding practices.
The prize: A massive win for readability and super easy bug hunting.

Lastly... to download ExceptionExtensions:

ExceptionExtensions for Java
ExceptionExtensions for JavaScript

Conditional evaluation in JavaScript - smarter better code

I have been trying to write some very concise code recently, that has a lot of conditional checks build into it. The problem with this, is the whole thing just becomes a long set of IF/ELSE statements, which is horrible to read.

Then I had a brainwave... JavaScript allows for the conditional evaluation of any expression.

so instead of writing this:

if(myCondition==true){
    doOtherFunction();
}

I can write this

    myCondition && doOtherFunction();

or even this

myCondition && myCondition>100 && doOtherFunction();

In fact, I can chain any number of expressions together, as long as they all evaluate to true.
and, it can even include assignment, as assignment is simply an expression.

myCondition && (myOtherVar=999);

And just to put the icing on the cake, I can combine this with Cast-to-Boolean.

Boolean(myCondition) && doOtherFunction();

or

!!myCondition && doOtherFunction();

Ok, the last example is getting a little extreme, but if you are looking for concise code...

A simple pad function in Javascript

I have been working on a small time tracking tool in angularJS, and I needed padding for strings in a digital clock. I had a quick search and although I found quite a few different implementations, none of them were very satisfactory.

So, I built one.

Quite simple really, but it will handle pretty much any type of input, and will pad both left and right.

Anyway, here it is.

function pad(input,size,paddingChar,direction){
    input = String(input);
    size = size || 1;
    paddingChar = paddingChar|| 0;
    direction = direction?String(direction).toUpperCase():"LEFT";
    var padString = Array(size).join(paddingChar);
    if(input.length < size){
        if(direction == 'LEFT'){
            input = (padString+input).slice(size*-1);
        }else if(direction == 'RIGHT'){
            input = (input+padString).slice(0,size);
        }
    }
    return input;
}

It is pretty easy to use:

To get Left padding:
pad(8,2,0) will return "08"
pad("234",5,0) will return "00234"
pad("top",5," ") will return "  top"

To get right padding:
pad("top",5," ","RIGHT") will return "top  "

If you like it, please feel free to use it (at your own risk, of course).

Fixing GVM

Sometimes GVM seems to disappear, so I am writing this to remind myself of the solution.

echo $GVM_INIT

unset GVM_INIT

[[ -s ~/.gvm/bin/gvm-init.sh ]] && source ~/.gvm/bin/gvm-init.sh

basically, clear the flag and re-initialise.

A better way to highlight Groovy in Intellij

Following my last post, on changing the Highlighting for Warnings, I have now discovered that I can upgrade the alert generated by the code inspector.

I still have the basic problem, that I am passing the wrong variable into a method call.
And I expect my IDE to highlight it as a serious error, not a warning.

So, goto:
Settings > Inspections
Groovy > Assignment issues
Incompatible type assignments

And change Warning to Error in the Severity options.


Now, if I make the same mistake, it is immediately highlighted in my code.

Changing Intellij Idea error & warning highlighting

I recently ran into a Method Signature Mismatch bug in some Groovy code, admittedly it was untested code, but even so, I would still expect my IDE to flag it as a major problem, rather than a weak maybe.

Unfortunately this seems related to the way Intellij detects problems, and because Groovy is dynamic, it is much harder to determine if something is a real error.

I would prefer things to be highlighted properly, which meant a few changes in Intellij highlighting settings.

Turns out, it is a little harder than I expected to get it to work out, ie: deep config change.
So I am adding a small screen capture to help.

1. Setting > Editor > Colors & Fonts > General
2. select Warning
3. Change colors to the ones you want to see
4. Verify it looks correct in the example code.

Please note, this only affects the code editor.
The files themselves will NOT get underlined.
ie: in Changes or Project views.

Betamax - a few notes

I am looking for a simple programmable intercepting proxy that can be used
to debug/modify requests to external systems.

It also has too work on legacy systems,
i.e.: older code, JVM, and tools.

I have built a simple prototype using JETTY,
but before I go too far, I want to evaluate anything else available.

I have been attempting to integrate Betamax into our current project to test it.

Here are a few of my notes, so I can pick it up later.

Note 1:
Betamax 1.1.2 is compiled using Java 1.7

How did I discover this the hard way:
if you get an error about TapeLoadException,
and nothing seems to make sense...
then put this line into your test....

throw new TapeLoadException("xxxx")

when it tries to run the code,
you get a standard version 51 error.
ie: it needs Java 1.7.


Note 2:
Ok, so I thought I would test it in Java 1.7,
even though I could never use it in the project...
Now I discover it seems to need Groovy 2
And we use Grails 2.1.0, which has Groovy 1.8

That's about enough for me.

Betamax looks like a useful tool, but without support for older tool chains,
it is going to be difficult getting it into a corporate project.


I have found a few 'similar' projects, as I find more, I will add them here.

wiremock
rest-driver

Most of these seem to be Record/Edit/Playback tools like VCR for Ruby.
To speed up functional tests, or make them work offline.
But even this limited functionality is a good thing.

Javascript compressors

Another note for myself...

I have been playing with Javascript compression for frameworks, notably ENYO.

I have recently run into a couple of issues, that I need to record.

1. unexpected 0x1a character at the end of files.
This seems to be caused by the windows COPY command, and can be fixed by specifying BINARY file format in the copy, by using /b in the options.

2. Unexpected Token ILLEGAL error message from the YUICompressor based compressor.
Turns out the error reported is quite removed from the cause.
I managed to get a better error message by using the Closure compressor from google, which reported that a missing semi-colon was causing a statement to overflow to the next line.

Finally, for static javascript file compression, I have settled on a new method.
Concatenate all the needed files into a single large file using CAT, save it, and then use Closure to compress it, again from the command line. This provides easy access to both input and output, so makes debugging easier.

Content-Length mismatch reported from Google App Engine

This is just a quick note to myself really.

I am building a java project on Google App Engine, using a local HOSTS override to point to my development server. After a recent round of refactoring, one of my pages stopped working in Chrome without any error messages.

I fired up Fiddler2 to investigate, and it reported:
Content-Length mismatch: Response Header indicated 4,021 bytes, but server sent 4,018 bytes.

This was rather strange and unexpected, as none of the changes messed with the headers.

Anyway, after spending rather too long reading through all of my filter code, I finally realized that I had removed the Sitemesh decorators from the page, but it was still being processed.

One quick change later and Sitemesh was validated as the culprit.

If you find yourself in a similar situation, I hope this is useful.

Exception.when, for more readable and maintainable code

I recently started using a small change in my Java coding that seems to improve readability, testability, and makes maintenance a lot easier.

Instead of using IF statements everywhere to check and then throw business exceptions, I now modify  my Exceptions and call "Exception.when(...)".

So this...

User user = userService.getUser(userId);
if(!user.isLoggedIn()){
 throw new MyBusinessException("User is not logged in");
}

becomes...

 
User user = userService.getUser(userId);
MyBusinessException.when(!user.isLoggedIn(),"User is not logged in");

It really is a lot easier to read the code when written like this.

The changes to your Exception class are easy too. All we are doing is hiding the conditional inside a static function. Just add a method like this...

static void when(boolean condition,String msg){
 if(condition){
  throw new BusinessException(msg);
 }
}

I am mostly using this with a RESTful system based on JAX-RS, and I have to do a lot of condition checking,which was becoming a nightmare to read with all those IF statements. Now, it reads like a dream, well, rather like proper English, so a perfect DSL.

FYI: I did look into everything I could think of to make it easier to implement, ie: generics, inheritance, etc. But you cannot use static methods in either interfaces or abstracts, and exceptions are blocked from generics. So we are stuck implementing this method in every Exception Class, but I think it is worth it.

Rooting a Momo11 Speed Android tablet

I just got my Momo11 Speed android tablet in the post.
It is rather flaky, and freezes a lot, so I am looking into it.

First things first...
roll back the firmware to 4.0.4...
but that needs a firmware backup...
which means rooting it.

So first task, root it.

I read around a bit and came across this blog post.
So risking unknown software, I installed ZhuoDaShi.
A bit more random button pressing (it is in chinese),
and it was rooted.

I verified this with Root Checker.