All posts by joekiller

I solve hard problems and have fun doing it.

Publishing Software

I published two pieces of software I had been working on recently. They are both related to consuming WebSocket and written in TypeScript.

Joe's Socket To Interface Utility
This utility is to help discover a common interface from object sent over
a WebSocket pipe.
Joe's WebSocket Manager
A simple interface to connect to a WebSocket and receive an event
when messages are pending.

The WebSocket manager has been used for some time in a side project and is very solid. The Socket To Interface Utility was a quick project I did yesterday to capture a single interface over a disparity of messages that start similar at the root but towards the leaves are very different.

I hope these are useful to someone. Enjoy!

Converting Streams of JSON Data to a Typescript Interface

If MariusAlch were on twitter, I’d @ him and say, “Thank you so much for your useful json-to-ts utility!”

Json to TS converts JSON to Typescript interfaces. It’s especially useful because it converts arrays of different sets to a unified interface. I used it to catch messages of varying payloads from Backpack.tf’s websocket and create a nice common interface for the messages. The payload are mainly homogeneous near the root of the object but it’s somewhat scattered towards the leaves. You can see from this example backpack.tf websocket payload how dense the data can be.

My utility does a deep examination of any messages’s structure and simplifies it to an abbreviated version which I called the “Shape” of the object.

type Shape = {
  [key: string]: string | null | boolean | 
                 number | Shape[]
} | string | null | boolean | number

You can see how the type is concise and sufficient for generic JSON message Interface detection. A sample of the tests are as follows.

expect(deepShape({a: "aValue"}))
        .toEqual({a: ""});
expect(deepShape({
                   a: "aValue",
                   numbers: [1,2,3]}))
       .toEqual({
                   a: "",
                   numbers: [0]});
expect(deepShape({a: true}))
        .toEqual({a: true});

While collecting messages, I take a hash of the Shape and, if I get a new sample, I add it to my collection. Once I’ve taken a sufficient sampling, the software assembles all the messages into an object that has my root type as the key and a vector to hold the samples. I pass the object to JsonToTs() and, “huzzah!”, out pops all my interfaces with optional types nicely intertwined.

const existing = 
  await fs.promises.readdir(shapePath);
const messages: {BPSocketEvent: Shape[]} =
  {BPSocketEvent: []}
for await (const file of existing) {
  const shapeFilePath =
    [shapePath,file].join(path.sep);
  const entry = 
    await fs.promises.readFile(shapeFilePath);
  messages.BPSocketEvent.push(
    JSON.parse(entry.toString())
  );
}
const interfaces = [];
for await (const typeInterface of JsonToTS(messages)) {
    interfaces.push(typeInterface);
}
console.log(interfaces.join('\n\n'));

So thank you very much MariusAlch!

Fix “ESLint: Unsafe call of an `any` typed value” using express (expressJS) and TypeScript

When trying to be diligent about keeping strict types via ESLint in a TypeScript based ExpressJS, You may hit the dreaded, “ESLint: Unsafe call of an any typed value. (@typescript-eslint/no-unsafe-call)” while trying to use the default express() method via:

import express from 'express';

const app = express();

After an npm i -D @types/express the error will remain. After searching around and seeing so many fixes for importing Request and Response types, I finally found an explanation of how to fix the any typed value error on the express() method via smichel17’s response to, “express() function and express namespace ambiguity #37920”. The nuance is in that “import” returns plain objects and “require” returns returns objects and functions. Note that you cannot destructure the “require” call so to get your “Request” and “Response” types without using “express.Request” and “express.Response” you still need the traditional import statement as well.

Update your code to the following and resolve the “Unsafe call of an any typed value” error without using the dreaded escape hatch of "noImplicitAny": false.

import express = require('express');
import { Request, Response } from 'express';

const app = express();
const port = 3000;

app.get('/', (req: Request, res: Response) => {
  res.status(200).send(
    'Express + TypeScript Server ' +
    'With Strict Types by Joe');
});

app.listen(port, () => {
  console.log(
    `⚡️[server]: Server is running ' + 
    ' at http://localhost:${port}`);
});

Windows Setup Load driver for NVMe m2 PCIe intel disk missing

TLDR: Switch your BIOS out of UEFI only and enable legacy boot options and legacy boot mode. Put the BIOS in classic mode etc. After install you can put it back to UEFI mode.

Background: Installing Windows on a new machine using UEFI mode only can result in Windows Setup not finding you disk or asking you to install a driver for your Intel Chipset and even if you happen to find it on Intel’s website, it won’t find the disk. It’ll say,

Load driver

No new devices drivers were found

No new devices drivers were found. Make sure the installation media contains the correct drivers, and then click OK.


For anyone who ran servers this’ll bring back broadcom raid driver hell flashbacks. The funny thing is I run Arch Linux and drivers have never been a problem in UEFI mode but Window’s Setup is an old dog it seems.

I had a hard time and even finally gave up and engaged Dell support. I flashed the drive with the latest firmware. Confirmed it works on another PC.  Dell support tried but they didn’t have this one in the database. Finally somehow I decided to try putting the BIOS back into classic mode and enabled legacy boot options and tried again and Windows saw the drive fine. I let Dell know how to fix it just in case someone else had it but then I saw the same issue on Twitter so i wrote this post. Hopefully no one else is in the dark on this.

Enable legacy boot options

 

In Loving Memory of Roselle Faulconer Scales

Roselle Faulconer Scales of Greenville, South Carolina, formerly of Fife, Virginia in Goochland County, died on January 13, 2020 at the age of 98.  The daughter of Mr. and Mrs. Aubrey Faulconer, she was born in Amherst County, Virginia on September 17, 1921, and graduated from Sweet Briar College.  Upon graduation, she took a job as a caseworker in Albemarle County’s Welfare Department.  There, in 1948, she married Joseph H. Scales, Jr., a World War II veteran and the love of her life.

In 1952, Roselle and Joe moved to Goochland County where they were active in many community activities.  Roselle was a founding member of the Goochland Fellowship and Family Services, now GoochlandCares, serving on its first board of directors.  She was a charter member of the Goochland Woman’s Club and the Goochland Historical Society, serving as board president for both organizations.  She also was a member and president of the Goochland Garden Club.  She served on the Elk Hill board, the Elizabeth Kates Foundation, and the Friends of the Goochland Library.  She volunteered for several years teaching financial management and parenting to lady prisoners prior to their release from the State Farm for Women.  She also served on and chaired the Goochland Regent in Service board.  In 2002, she was honored to be Goochland’s Christmas Mother.

After retiring from 22 years of teaching, Roselle joined the Point of Fork Chapter of the DAR and served as its vice-regent.  The Huguenot Society of the Founders of Manakin made her an honorary member.

Roselle was an elder of historic Byrd Presbyterian Church.  When the church celebrated its 250th anniversary, she, at the request of the session, wrote a brief history of the church from sessional minutes and other sources, a copy of which can be found in the library at Union Theological Seminary and the Presbyterian Historical Society in Montreat, North Carolina.

In 2005 when they were 82, Roselle and Joe moved to Greenville, South Carolina to be near their daughter, Mary.  At that time, the Goochland County Historical Society and Goochland Garden Club honored them for their 50 years of service to the county.  Joe died in 2008.  Roselle embraced her life in Greenville as she had in Goochland.  She made many friends and was active in Westminster Presbyterian Church. She also enjoyed being part of a garden club that no longer gardened, but just had lunch; a tennis club that no longer played tennis, but just had lunch; a book club that read occasionally, but visited regularly and had lunch; and a bridge club that did, indeed, play bridge.  Her life was full and friend-filled to the end.

Roselle was predeceased by her husband, her sister Mildred Faulconer Bryant, and her brother Aubrey Faulconer, Jr.  She was also predeceased by a beloved daughter, Grace Scales Yoder.  She is survived by daughter Mary Scales Lawson and her husband Dr. Jeffrey Lawson of Greenville, South Carolina, son-in-law Dr. Joseph W. Yoder and his wife Patty Yoder of Oakton, Virginia.  Other family members include grandchildren Roselle Lawson Bonnoitt, Margaret Lawson Meadows, Joseph Scales Lawson, Caroline Alexander Yoder, Madeline Yoder Schmidt, Grace Amanda Yoder and seven great-grandchildren.

The family wishes to thank Waterstone on Augusta and the many caregivers at BrightStar for their loving care of Roselle in her last year.

A memorial service will be held at Westminster Presbyterian Church on Sunday, January 19 at 2 pm. A funeral and burial will be held at Byrd Presbyterian Church on Saturday, January 25 at noon.  In lieu of flowers, the family asks that memorials be sent to Westminster Presbyterian Church, Byrd Presbyterian Church, or a charity of one’s choice.

Rest in peace Grandmother.

Roselle Scales

aws-cdk template porting migration tips

AWS-CDK Migration Tips

When migrating to a new framework there are going to be some working pains. This is just a collection of frustrations that I encountered while adopting aws-cdk. I did some research prior and found the article “Hey CDK, how can I migrate my existing CloudFormation templates?” by Philipp Garbe and the “core module AWS CDK” documentation most helpful in thinking about migrating initially.

Import Immutable Roles

Use the mutable flag when importing existing roles with Role.fromRoleArn() otherwise the precision of the aws-cdk may lead to the dreaded “Maximum policy size of 10240 bytes” error. Eventually aws-cdk issue #4465 will be fixed and we will welcome the precise IAM policies the CDK generates.

The maximum policy size error was most often encountered on CodePipeline deploy roles where we had a large number of independent artifacts deploying CloudFormations.

Explicit to_string() in python

Having to explicitly call the core.Fn.get_att(‘Foo’, ‘Bar’).to_string() operator instead of using a str() for f'{var}’ style tripped me up.

I noticed in my IDE that the signature called for a string (thanks types!) so I tried:

str(core.Fn.get_att('Foo','Bar'))

and

f'{core.Fn.get_att('Foo','Bar')}'

but because only __repr__ is defined in the python interface I got an ugly object name when I expected __str__ to be implemented. I overlooked the to_string(), which is a pretty common method for many object oriented languages, as I expected the class to behave more pythonically.

Beware of Copy Paste / Naming

Name a stack the same as another? You get a diff but if you aren’t paying attention you’ll blow away a stack before realizing it. Also you end up with the old stack as well not updated because of this config SNAFU.

Import Pains

CfnImport is great for importing old CloudFormations but the stack is immutable upon import. Any changes to the stack must happen prior to making the call. We leaned on PyYAML but then had to undo a few of the niceties of only processing the template with AWS based systems.

Intrinsic Function Shortcuts

For example all bang, “!” ie “!Ref” or “!Sub”, references need to be updated to be the full function command, ie “Ref:” and “Fn::Sub:”.

Attributeerror: ‘datetime.date’

IAM Policy Documents specifying the Version unquoted instead of as a string, ie Version: 2012-10-17 instead of Version: ‘2012-10-17’, will have the cdk synth command greet them with following obscure error.

This error also occurs on AWSTemplateFormatVersion blocks so beware.

 AttributeError: 'datetime.date' object has no attribute '__jsii__type__'.

Example CfnImport

Here is an example of using the CfnImport to inject parameters into a traditional template and then load it into the CDK stack.


import yaml
from aws_cdk import core
class RawStack(core.Stack):
def __init__(self, scope: core.Construct, name: str, template_path: str, wrapped_parameters=None,
**kwargs) -> None:
"""import a stack off a path and munge in ssm variables if desired
:param template_path: path to raw stack being imported
:param wrapped_parameters: map of Parameter keys and default values
:param kwargs: all the stack stuff
"""
super().__init__(scope=scope, name=name, **kwargs)
if not wrapped_parameters:
wrapped_parameters = {}
template_path = Path(template_path)
with open(template_path, 'r') as f:
template = yaml.load(f, Loader=yaml.SafeLoader)
if project_name:
for pk, pv in template['Parameters'].items():
if 'Default' in pv:
if pk in wrapped_parameters:
template['Parameters'][pk]['Default'] = wrapped_parameters[pk]
elif pv['Default'] == "":
template['Parameters'][pk]['Type'] = 'AWS::SSM::Parameter::Value<String>'
template['Parameters'][pk]['Default'] = str(pk)
else:
if pk in wrapped_parameters:
template['Parameters'][pk]['Default'] = wrapped_parameters[pk]
else:
template['Parameters'][pk]['Type'] = 'AWS::SSM::Parameter::Value<String>'
template['Parameters'][pk]['Default'] = str(pk)
core.CfnInclude(self, 'RawStack', template=template)

view raw

raw_stack.py

hosted with ❤ by GitHub

raw_stack.py gist link

 

Fixing unhandled instruction bytes error Running Valgrind on AWS CodeBuild

When running Valgrind against one of our C libraries we encountered some discrepancies in the build where locally all would pass but on AWS CodeBuild using the aws/codebuild/standard:2.0 image we would get errors like:

vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0x7D 0x48 0xEF 0xC0 0xC5 0xF9 0x2E 0x45

The full message was like:

==16128== Memcheck, a memory error detector
==16128== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==16128== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==16128== Command: /root/build/meh/test/.libs/state
==16128== 
[==========] Running 2 test(s).
[ RUN      ] test1
vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0x7D 0x48 0xEF 0xC0 0xC5 0xF9 0x2E 0x45
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==16128== valgrind: Unrecognised instruction at address 0x1173bd.
=...
==16128== Your program just tried to execute an instruction that Valgrind
==16128== did not recognise.  There are two possible reasons for this.
==16128== 1. Your program has a bug and erroneously jumped to a non-code
==16128==    location.  If you are running Memcheck and you just saw a
==16128==    warning about a bad jump, it's probably your program's fault.
==16128== 2. The instruction is legitimate but Valgrind doesn't handle it,
==16128==    i.e. it's Valgrind's fault.  If you think this is the case or
==16128==    you are not sure, please let us know and we'll try to fix it.
==16128== Either way, Valgrind will now raise a SIGILL signal which will
==16128== probably kill your program.

The error seems to indicate that the architecture doesn’t seem to match what the docker image had so going off of a Linux Headers Reinstall article we added the following and then the architecture packages were fine.

apt upgrade --fix-missing -y && apt autoremove -y && apt autoclean -y

 

AWS CDK CLI can only be used with apps created by CDK error

I upgraded my AWS CDK to 1.10.1 today because it prompted me via:

**************************************************
*** Newer version of CDK is available [1.10.0] ***
*** Upgrade recommended                        ***
**************************************************

After doing the upgrade via

npm install -i -g aws-cdk

I went to do a cdk ls or cdk diff and was greeted with the error:

CDK CLI can only be used with apps created by CDK >= 1.10.0

Googling around wasn’t too helpful but finally I figured out that it was complaining that my python dependencies  had the old aws-cdk libraries installed.

A quick

rm -r .env/
python -m venv .env
pip install -r requirements.txt

And I was back in business

cdk ls
integration-pipeline

ZyXEL EMG3425-Q10A Port Forwarding

The ZyXEL EMG3425-Q10A has NAT Port Forwaring but it doesn’t seem to work well because the Remote Management section has been patched out. This causes the remote management screen to always boot on the IP that is the same as the Port Forwarding Default Server Setup. You need to do two things to fix this mess. First change the default server setup to be the target you normally will port forward to. Turn on a firewall on this host. Second, you need to forward the WWW and HTTPS rules to that host.

When adding additional rules, click “add” instead of apply. Add ports like 25565 to host minecraft or 27015 to TF2.

You cannot delete rules or you have to re-enter all of them in the right order. AGAIN.

EDID Reading on Arch Linux

There are several tools to read the Extended Display Identification Data, EDID, from systems but I found LinuxTV’s edid-decode the most thorough when debugging for a linux 5.0.x display boot flicking problem.

On arch I ran installed edid-decode-git and then ran a quick script:

for f in `find /sys/devices -name 'edid'`; do sudo cat $f| edid-decode;done

and I got something like:

EDID version: 1.4
Manufacturer: BOE Model 65a Serial Number 0
Made in week 1 of 2015
Digital display
6 bits per primary color channel
DisplayPort interface
Maximum image size: 34 cm x 19 cm
Gamma: 2.20
Supported color formats: RGB 4:4:4, YCrCb 4:4:4
First detailed timing includes the native pixel format and preferred refresh rate
Display x,y Chromaticity:
  Red:   0.6416, 0.3437
  Green: 0.3183, 0.6103
  Blue:  0.1494, 0.0439
  White: 0.3125, 0.3281
Established timings supported:
Standard timings supported:
Detailed mode: Clock 139.770 MHz, 344 mm x 194 mm
               1920 1968 2000 2080 hborder 0
               1080 1083 1089 1120 vborder 0
               +hsync -vsync 
               VertFreq: 59 Hz, HorFreq: 67197 Hz
Detailed mode: Clock 111.820 MHz, 344 mm x 194 mm
               1920 1968 2000 2080 hborder 0
               1080 1083 1089 1120 vborder 0
               +hsync -vsync 
               VertFreq: 47 Hz, HorFreq: 53759 Hz
ASCII string: J125V
Manufacturer-specified data, tag 0
Checksum: 0xa9 (valid)

This helped when trying to diagnose: black screen on Dell XPS 15 with kernel 5.0 and Bug 109959 – REGRESSION: black screen with linux 5.0 when starting X