Sharing session data between Rails and Nodejs

In a previous post I wrote about securing my NodeJS service by enforcing authentication using a session token. Now there was another problem: the node service was vulnerable to identity spoofing because the client tells the service who he is, and the service will always trust him. This is bad.

Fortunately, the Rails app uses Redis as its session store and it already stores the user id in there. So the only thing that left to be done is for Node to read the session data out of Redis.

After a few hours of experimentation and looking around, I learned a few things:

  • Rails serializes the session data before storing it
  • Rails uses the Marshal module to serialize session data
  • The result of serialization is not exactly plain text, which explains some gibberish characters I saw when I print it out in redis-cli / nodejs
  • Marshal is version sensitive
  • There is a nodejs library node-marshal to parse the Ruby Marshal output
  • There is a monkey patch to change the default from marshal to JSON
  • There is another nodejs library called marsha, but for whatever reasons it didn’t work for me

So with node-marshal I was able to deserialize the Rails session data and read it in nodejs, closing the door on spoofing attacks.

 

 

Advertisements

Questrade.js updated with example

I am a believer of the couch potato investment strategy: choose asset allocation and stick to it and periodically rebalance. I am managing my accounts and also my wife’s accounts: there are going to be 7 accounts in total (2x RRSP, TFSA,non-registered accounts, plus one spousal RRSP). I am treating the accounts as one family portfolio. Rebalancing is going to be painful and I am not going to do it unless it is easy! Hence I updated my Questrade API nodejs wrapper to include a simple script to list position of all the accounts. For example:

SYMBOL CURRENT MARKET VALUE OPEN QUANTITY CURRENT PRICE
AVO.TO 2996 200 14.98
VIU.TO 13769.2 580 23.74
VXC.TO 8625 300 28.75

See example/portfolio_checker for code.

This is the first step in rebalancing my family’s portfolio. The next step is to take target asset allocation percentages as input and then actually calculate how much to buy and sell each stock.

 

Questrade API nodejs wrapper

I wanted to write a script using the Questrade API to help me rebalance my portfolio. I guess wrapping the rest calls behind a function wrapper is a good start. Here is what I started, feel free to help me out, or just take a look:

https://github.com/dk1027/questrade.js

questrade.js is the wrapper around the REST API. When you instantiate it, pass the result of redeeming the refresh token to the constructor, as shown in questapp.js.

I want to experiment with the API initially so I wrote questapp.js. What I do is load it in the node repl and call functions in questrade.js. Something like this:

ltse@ltse-ubuntu:~/questrade_sample$ node
> .load questapp.js
> ...noise ommitted...
> q.accounts()
undefined
> { accounts: 
   [ { type: 'Margin',
       number: '12345678',
       status: 'Active',
       isPrimary: true,
       isBilling: true,
       clientAccountType: 'Individual' },
     { type: 'TFSA',
       number: '2345678',
       status: 'Active',
       isPrimary: false,
       isBilling: false,
       clientAccountType: 'Individual' } ],
  userId: 34567 }

undefined
> 

Come to think of it, wouldn’t it be nice if there is something that would auto-generate functions base on a rest api definition? Someone probably has came up with it already…

Update:

Questrade.js updated with example

Questrade API sample code with nodejs and request

Questrade released an API to retrieve account information, market data, and even making trades. That’s awesome! They have helpful documentation and even a developer console to make the API calls. Here is some sample code to use their API. Remember to paste your own refresh token (their Getting Started page provides instructions on how to generate it)

var request = require('request');

var refresh_token='YOUR TOKEN HERE';

// This gets us the access token for authorizing subsequent requests
console.log("getting access token");
request.get(
	{url:'https://login.questrade.com/oauth2/token?grant_type=refresh_token&refresh_token=' + refresh_token},
	function(err, response, body){
		if(err){
			console.log(err);
			return;
		}

		console.log(response.statusCode);

		if(response.statusCode != 200){
			console.log(body); // If we get here, it is likely to be a 'bad request' because we are using the same refresh_token more than once
			return;
		}

		b = JSON.parse(body)
		var access_token = b.access_token;
		var api_server = b.api_server;

		// Let's try to get our account information
		if(response.statusCode == 200)
		{
			console.log('Make a request');
			request.get(
				{
					url: api_server + 'v1/accounts',
					auth:{
						bearer : access_token
					}
				},
				function(err, response, body){
					if(err){
						console.log(err);
						return;
					}
					console.log(response.statusCode);
					console.log(body);
				}
			);
		}
	}
);

This code redeems the refresh token for an access token, and then use the access token to make a get request to v1/accounts to get account information. You should be able to reuse the access token for other requests; however you can only use the refresh token once. If you use it more than once, you will get a 400 bad request.

Mesos framework example in Python

The example I am presenting here is built on this blog post by James J Porter. James’ example runs a shell command to echo Hello World. Now, let’s say we want to do some computation and the framework needs to tell the slaves to get the binaries/data from somewhere. How do we do that?

The answer lies in the mesos.proto file:

/**
 * Describes a command, executed via: '/bin/sh -c value'. Any URIs specified
 * are fetched before executing the command.  If the executable field for an
 * uri is set, executable file permission is set on the downloaded file.
 * Otherwise, if the downloaded file has a recognized archive extension
 * (currently [compressed] tar and zip) it is extracted into the executor's
 * working directory. This extraction can be disabled by setting `extract` to
 * false. In addition, any environment variables are set before executing
 * the command (so they can be used to "parameterize" your command).
 */
message CommandInfo {
  message URI {
    required string value = 1;
    optional bool executable = 2;
    optional bool extract = 3 [default = true];
  }

Mesos take cares of downloading and extracting the files; all we need is state where to get it from. Therefore compose the task message like this (in Python):


task = new_task(offer)
uri = task.command.uris.add()
uri.value = "path-to-file"
task.command.value = "command to-run-something"

What exactly do we put as the uri? Looking at the Mesos code here, turns out the URI can point to a local file, HDFS, http, ftp, etc.

Base on these new found knowledge, I put bells and whistles on the hello world framework (actually I took the hello world part away). The example framework here is going to tell the slaves to talk to a web service to get a Python script, use the script to sum up a few numbers, then send the results to the web service. The web service is going to print out the results. When it is done, the framework will stop.

Continue reading

Read protobuf messages in Nodejs / JavaScript

I have a SQLite database that stores serialized protobuf messages. I need an easy way to look at what’s in these messages. Reading the protobuf messages with JavaScript seems a good choice for me because I can see everything in JSON.

Here is how to read a protobuf message in JavaScript and display as JSON:

ProtoBuf = require('protobufjs');
sqlite3 = require('sqlite3').verbose();

// Read the proto file
var builder = ProtoBuf.loadProtoFile("/path/to.proto");
JS = builder.build("my_proto_package");

// Open SQLite database, since that's where my protobuf messages are stored
var db = new sqlite3.Database('my_sqlite.db');
db.each("SELECT proto as p FROM tbl LIMIT 100", function(err, row){
	var msg = JS.MyMessage.decode(row.p);
	console.log(msg);
});

row.p is just a buffer. So if you have the protobuf message in a file, you can do this instead:

require('fs')
fs.readFile("file", function(err, data){
	console.log(JS.MyMessage.decode(data));
});

Do see the ProtoBuf.js example, it is very helpful: https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/protoify