Category Archives: Software

Publishing Software

I published two pieces of software I had been working on recently. They are both related to consuming WebSocket and written in TypeScript.

Joe's Socket To Interface Utility
This utility is to help discover a common interface from object sent over
a WebSocket pipe.
Joe's WebSocket Manager
A simple interface to connect to a WebSocket and receive an event
when messages are pending.

The WebSocket manager has been used for some time in a side project and is very solid. The Socket To Interface Utility was a quick project I did yesterday to capture a single interface over a disparity of messages that start similar at the root but towards the leaves are very different.

I hope these are useful to someone. Enjoy!

Converting Streams of JSON Data to a Typescript Interface

If MariusAlch were on twitter, I’d @ him and say, “Thank you so much for your useful json-to-ts utility!”

Json to TS converts JSON to Typescript interfaces. It’s especially useful because it converts arrays of different sets to a unified interface. I used it to catch messages of varying payloads from Backpack.tf’s websocket and create a nice common interface for the messages. The payload are mainly homogeneous near the root of the object but it’s somewhat scattered towards the leaves. You can see from this example backpack.tf websocket payload how dense the data can be.

My utility does a deep examination of any messages’s structure and simplifies it to an abbreviated version which I called the “Shape” of the object.

type Shape = {
  [key: string]: string | null | boolean | 
                 number | Shape[]
} | string | null | boolean | number

You can see how the type is concise and sufficient for generic JSON message Interface detection. A sample of the tests are as follows.

expect(deepShape({a: "aValue"}))
        .toEqual({a: ""});
expect(deepShape({
                   a: "aValue",
                   numbers: [1,2,3]}))
       .toEqual({
                   a: "",
                   numbers: [0]});
expect(deepShape({a: true}))
        .toEqual({a: true});

While collecting messages, I take a hash of the Shape and, if I get a new sample, I add it to my collection. Once I’ve taken a sufficient sampling, the software assembles all the messages into an object that has my root type as the key and a vector to hold the samples. I pass the object to JsonToTs() and, “huzzah!”, out pops all my interfaces with optional types nicely intertwined.

const existing = 
  await fs.promises.readdir(shapePath);
const messages: {BPSocketEvent: Shape[]} =
  {BPSocketEvent: []}
for await (const file of existing) {
  const shapeFilePath =
    [shapePath,file].join(path.sep);
  const entry = 
    await fs.promises.readFile(shapeFilePath);
  messages.BPSocketEvent.push(
    JSON.parse(entry.toString())
  );
}
const interfaces = [];
for await (const typeInterface of JsonToTS(messages)) {
    interfaces.push(typeInterface);
}
console.log(interfaces.join('\n\n'));

So thank you very much MariusAlch!

What the heck have I been doing?

I haven’t made much effort to blog recently so I figured I would do a catchup article on some of the projects I’ve worked on recently.

gor is a http traffic replay tool written in go. When I first found Gor, it was a great tool but there were several bugs that made it unstable or unreliable. When I had a little time, I submitted several enhancements.
  • https://github.com/buger/gor/pull/116 added dynamic http worker scaling which was key for throughput.  I also provided documentation updates to help others discover and identify bottlenecks in their testing systems.
  • https://github.com/buger/gor/pull/94 sorted the TCP packets properly which was pretty crucial for replaying connections that had out of sequence packet arrival. Alejandro Martinez of Vivid Cortex helped me in spotting the problem.

moto is a mock AWS endpoint written in Python. We use AWS a ton at Room Key and needed to better test our internal Python tools. Moto had a good feature set but didn’t quite implement the more advanced things we were using so I added to the library over 11 different PRs.  Here are a few of the biggest features I added:

logstash-kafka and jruby-kafka are my Ruby projects that I created to make the Kafka plugin for Logstash. The following link is where the Elasticsearch blog announces support for Kafka in Logstash and they credit my project as being the origin of the support, https://www.elastic.co/blog/logstash-1-5-0-beta1-released. Because 1.5 isn’t out yet, I still maintain support for my independent plugin and provide input and support otherwise for the official plugins logstash-output-kafka and logstash-input-kafka.

 Sometimes other projects just aren’t getting enough attention for their PRs from their primary owner and I occasionally will volunteer to be a maintainer (as I want my PR’s merged). They are generally low volume projects that don’t need much love beyond a merge here and there combined with a release. These include:
  •  s3_file, a Chef resource for downloading files from S3 for Chef recipes
  • grunt-aws-sdk, a wrapper for AWS’s Javascript SDK for Grunt tasks.

I’ve touched quite a few different projects over the last year or so. In many cases I consider these projects to be an act of standing on the shoulder of giants and adding a little extra to each. That being said, even as I contribute to these external projects, most of my work happens internally at Room Key. You can see the difference between my public activity graph versus my private graph:
Screen Shot 2015-05-07 at 11.02.16 PM Screen Shot 2015-05-07 at 11.02.33 PM
So there you have it.  A small snapshot into what’s been happening over the last year.