All posts by joekiller

I solve hard problems and have fun doing it.

What the heck have I been doing?

I haven’t made much effort to blog recently so I figured I would do a catchup article on some of the projects I’ve worked on recently.

gor is a http traffic replay tool written in go. When I first found Gor, it was a great tool but there were several bugs that made it unstable or unreliable. When I had a little time, I submitted several enhancements.
  • added dynamic http worker scaling which was key for throughput.  I also provided documentation updates to help others discover and identify bottlenecks in their testing systems.
  • sorted the TCP packets properly which was pretty crucial for replaying connections that had out of sequence packet arrival. Alejandro Martinez of Vivid Cortex helped me in spotting the problem.

moto is a mock AWS endpoint written in Python. We use AWS a ton at Room Key and needed to better test our internal Python tools. Moto had a good feature set but didn’t quite implement the more advanced things we were using so I added to the library over 11 different PRs.  Here are a few of the biggest features I added:

logstash-kafka and jruby-kafka are my Ruby projects that I created to make the Kafka plugin for Logstash. The following link is where the Elasticsearch blog announces support for Kafka in Logstash and they credit my project as being the origin of the support, Because 1.5 isn’t out yet, I still maintain support for my independent plugin and provide input and support otherwise for the official plugins logstash-output-kafka and logstash-input-kafka.

 Sometimes other projects just aren’t getting enough attention for their PRs from their primary owner and I occasionally will volunteer to be a maintainer (as I want my PR’s merged). They are generally low volume projects that don’t need much love beyond a merge here and there combined with a release. These include:
  •  s3_file, a Chef resource for downloading files from S3 for Chef recipes
  • grunt-aws-sdk, a wrapper for AWS’s Javascript SDK for Grunt tasks.

I’ve touched quite a few different projects over the last year or so. In many cases I consider these projects to be an act of standing on the shoulder of giants and adding a little extra to each. That being said, even as I contribute to these external projects, most of my work happens internally at Room Key. You can see the difference between my public activity graph versus my private graph:
Screen Shot 2015-05-07 at 11.02.16 PM Screen Shot 2015-05-07 at 11.02.33 PM
So there you have it.  A small snapshot into what’s been happening over the last year.

Ubuntu 12.10 and 12.04.1 LTS release note snippets

With the release of Ubuntu 12.10 and 12.04.1 LTS to the world, Ubuntu is showing their preference for Tomcat 7, Java 7 and Python 3.  Check out some highlights of the release below.

12.04.1 PrecisePangolin

12.10 QuantalQuetzal

12.04.1 Highlights:
Please note that Tomcat 7 will replace Tomcat 6 as the supported version of Apache Tomcat in Ubuntu 12.10.
Please note that OpenJDK 7 will become the default Java implementation in Ubuntu 12.10.
Chef 0.8.16 (948437) and corresponding Ohai packages (948438) removed from precise at the request of OpsCode.
12.10 Highlights:
Chef 10.12.0 has been re-introduced to Ubuntu in-conjunction with OpsCode.
canonical urges people to migrate to python 3: If you have your own programs based on Python 2, fear not! Python 2 will continue to be available (as the python package) for the foreseeable future. However, to best support future versions of Ubuntu you should consider porting your code to Python 3. Python/3 has some advice and resources on this.

Ubuntu 12.10 ships OpenJDK7 as the default Java implementation. This brings improved performance, new features and better compatibility with other Java 7 implementations.

Use of the OpenJDK6 is now deprecated and the openjdk-6-* packages in universe for Ubuntu 12.10 will not be provided in future releases of Ubuntu.

Install Firefox on Amazon Linux x86_64 Compiling GTK+

Amazon Linux doesn’t offer the Gimp Tool Kit (GTK+) so if you want to run Firefox on an Amazon Linux system, say for Selenium testing, you are left having to compile the system yourself.  Luckily you have found this post.  Create the script below, run it as root and it will build all the components needed for GTK+ and its dependencies for Firefox to run just fine on the system.

vi ./gtk-firefox
chmod 755 ./gtk-firefox
sudo ./gtk-firefox

After you have built the packages, add the /usr/local/bin to your path by updating your .bashrc file.

cat << EOF >> ~/.bashrc
export PATH

Here is the gtk-firefox file for your pleasure.

If you are running OSX Mountain Lion or above and cannot get Firefox to run via the SSH -X command, make sure you have XQuartz installed as Apple removed X11 by default.

Edited to make Firefox latest release more reliable. Updated with Gist.

Edit 11/21/2012: Added dbus-glib dependency to gist. Added notes about running on OSX

Creating a X.509 or Signing Certificate for AWS EC2 using Powershell and Windows SDK

Currently Amazon AWS only allows Base-64 encoded certificates to be used as an EC2 credential.  Further when creating a user in IAM, Amazon doesn’t provide a convenient certificate generator which it does allow for the root user.  If you want to create these type of certificates on Windows you will find that it is not easy to get the certificate out of a binary (DER) format.  Many will point you to OpenSSL to do the conversion and that is fantastic however some may not be able to use OpenSSL.

I am going to lay out some steps that will help you quickly create an X.509 certificate and private key using the Windows SDK makecert.exe utility and Powershell.

First download the Windows SDK.  When installing, only the Tools option is necessary.  Usually the SDK installs to C:\Program Files\Windows SDK\version\bin.  I would suggest that you modify your path to include the SDK bin directory if you are going to make a lot of these certificates.  These instructions assume that makecert is in your path.

Makecert has a number of functions, but the feature we are interested in is its ability to generate self signed certificates with a straightforward command.  All certificates output are in a DER binary format so they are currently unsuitable for AWS consumption.  We will use powershell to convert from a binary object to a Base-64 string.  Note that makecert normally creates a single file containing both the private key and the public key.  Since we want these elements in separate files, we use the -sv toggle which saves the private key to a .pvk file.  One last gotcha to note is that the tool seems to want you to specify the resulting files with the extensions as show in the help and examples.  If you don’t use the .pvk and .cer extensions it might not output the file.

Assuming that you have the SDK install and can run makecert, here are the steps to get your certificate AWS ready.

Create the self signed certificate and corresponding private key file using makecert:

makecert -sv privatekey.pvk certificate.cer

Next we are going to use powershell and some .NET magic to process the binary files into a text friendly BASE-64 format (PEM).

Process the certificate first:

[byte[]] $x = get-content -encoding byte -path .\certificate.cer

[System.Convert]::ToBase64String($x) > .\cer-ec2creds.PEM

Next Process the private key:

[byte[]] $x = get-content -encoding byte -path .\privatekey.pvk

[System.Convert]::ToBase64String($x) > .\pk-ec2creds.PEM

You can now examine the resulting files in notepad to confirm that they are indeed in a BASE-64 format.

notepad .\cer-ec2creds.PEM

notepad .\pk-ec2creds.PEM

The files should work fine even if they are missing the proper headers and footers.  If you want to include them, they should be as follows.  Remember to add an end line character to the file as well.

For the certificate PEM file:



For the private key PEM file:



I hope this is useful.  Please feel free to comment and share other methods.

Here are some references:


Financial Planning Shortcuts and Freedom from Stress

I have created a simple spreadsheet to reduce the stress of planning and keeping on track with a budget.  I have found that I cannot stick to a rigid budget, but if most of my financial moves are automatic I am much more likely to keep on the budget.  The purpose of this sheet is to allow me to see a number of details of my financial life.  I used to self calculate these values each year or so after a pay adjustment to make sure the numbers were right.  Finally, I just built a Google doc out of it.

The sheet is useful to me in a number of ways such as:

  • making sure that my spending is in balance with the 50-20-30 net income budgeting principal,
  • being able to quantify what amount of money I can spend at my discretion each day,
  • allowing me to calculate recurring costs to allocate for automatic bill pay,
  • proving visibility into how much I am paying for services daily, monthly and yearly,
  • set magic totals which give savings goals and ideas as to where I want my reserve funds to be,
  • and of course play around with salary changes to see how it affects the end game.

The 50-20-30 budget has been around for some time and basically says I want to spend 50% on needs, 20% on savings, and my 30% on wants.  I turn this rule on its head slightly because I find pinpointing every percentage of my spending impossible.  Instead of apply the percentages against all recurring points of the budget. It helps me live within my means and still have some discretion as to how I’m going to live and spend money day to day.  Basically this is structure to allow me to be as free as possible from financial angst.  If you want your life to be a little more stress free, automating your budget with these principals can go a long way.

I spend a maximum of 50% of my net income on absolute needs such as power, rent, water, car insurance, gasoline and taxes.   Having a place to live with running water and electricity are needs.  A car can be considered a need or a want depending on the circumstance of your income.  Is a car your only means to get to work every day?  You should chalk the car, maintenance, gasoline and insurance up as a need for the time being.

Next, 20% of the net income Savings should be on a well diversified split between long term retirement vehicles, stock and mutual fund investments and more liquid options such as a savings account or bank CDs.  Keep in mind that with 401k contributions you make pretax but should count it in with the savings percentage.  How you split up savings should reflect your risk style.   Since I am a younger investor, I am trying to max out my Roth IRA prior to maxing out the 401k as my salary will most likely increase over the years thus, theoretically, maximizing the tax savings as Roth IRA withdrawals are tax free.  I find stocks to be the last thing I am investing in and see them more as hobby but you can get a good return.  Remember the day you want to buy is the day everyone else sold because it hit rock bottom.  If you believe the company will be around after the bad days, take the investment and run with it.  Keeping your savings automatic is the key to maintaining the percentage.

Finally there is the 30% wants budget.  Some people like to list the everything as the 50-30-20 budget, but needs and savings come first.  Wants are the discretionary.  They are the creature comforts and services you choose to use.  You must be serious about identifying something as it really is.  Cable, Internet or cigarettes are not needs, no matter how much you want them.  Note that these are all recurring wants.  The key to the spreadsheet is to list as many things that you pay for over and over as possible that you see as something you want but are not planning to cut off.  You could include things like cellular phone, magazines, donations to organizations, Internet services, recurring software upgrades, credit card fees, travel budget and really anything you wish to plan for and use.  Wants are what you want right?

The goal of the budget by percentages is to identify as many consistent, recurring costs as possible and quantify them against a yearly plan.  If your income is unknown month in in and month out, the sheet can still help you plan your recurring needs and wants.  Try your best to estimate your income.  If you have a salary or somewhat constant income the sheet is even more powerful for giving you freedom.

Luckily or unluckily we have many recurring costs within our lives.  By mastering our recurring costs, we are able to be more conscience in our ability to tackle unknown costs.  By listing all recurring costs into the sheet, you can finally come down to your Left Over Number.  The Left Over is the money left over after you have saved a percentage of your income, you have accounted for all your recurring needs and you have indulged in a number of recurring wants or services.  This should be your money in the pocket number.  It is a special number as it cuts down to give you a Cash Per Day number.  If you budget is out of whack, it could be a depressing but in the end, this is the money in the pocket number.

Finally the key to all this savings and cash flow business is to make it as automatic as possible.  By identifying how much money you need to work through a month, you are releasing yourself from the stress of forgetting to pay bills.  Further, you are more conscience with you day to day expenses.  I setup two bank accounts where one is for the left over want money and another is for needs and recurring wants that I was planning to keep.  Each paycheck the Move To Bill Per Check amount in the spreadsheet is moved to a separate checking account.  Then I only have available a limited pool of spending money and my automatic bills are free of overdrafts.  In the end I had to go to a weekly cash allowance and when times get really bad, it is a daily cash allowance.  If I spend too much, I cut to daily allowances until I have a buffer for another week.  Buffers are your friend.  Build the cushion if you can and feel a little more free.

I hope you find this post useful.  Below is the link to the sample Google doc which you can use for your own planning.

View Joe’s Sample Financial Planning Google Doc

Testing iexplore with Selenium Server as a Jenkins/Hudson Slave via Seleniumhq Plugin

Selenium Server, the 2.0 blend of Selenium RC and Webdriver, is the latest in CI Testing goodness from the Selenium project and SeleniumHQ.  During my experimenting with trying to get Selenium to take scripts made in the Selenium IDE and run them with the new selenium-server-standalone-2.0.0.jar via the Jenkins Seleniumhq Plugin on a Jenkins/Hudson slave, I had a few different issues.  My primary problem was getting *iexplore tests to execute from a Jenkins/Hudson slave node.  The slave is running as a service started as a domain user instead of Local Service.  The slave has to run as a domain user because the Jenkins slave is also doubling as a Windows build server running off a Linux master.  The goal was to test with Firefox 5 and Internet Explorer 7 in a Windows Server 2003 R2 x64 environment.  In the end, I could only get *iexplore tests to run reliably by using Window’s automatic logon and then launch the Hudson/Jenkins slave as a startup shortcut which was just:

javaws http://hudsonhost.local:8080/hudson/computer/slave-agent.jnlp

I believe this will also work with nearly any other Windows distribution up to the latest 7/2008R2 series. It was undesirable to run the Slave service in this way however it may just be what is necessary to test older software with a Windows Server 2003 platform.  This approach locks you out of the console of the server, but you can leave the user with just user privileges and then remote in to administer if needed.

By the way, Firefox 5 ran flawlessly as a domain user after creating a Firefox profile for Selenium.

Welcome Nome! Re: Alaska Dot Experiment

My friends,

I announce a milestone for our experiment.  The Alaska post is a success!  Judging by the Clustrmap below, we have successfully lured 1 – 2 people to click on this site and see what the heck I am spouting about!   I’d like to thank everyone who contributed to this including Crazyredbeard who was here to inspire the post and especially that or those Alaskan(s) who went for the sweet sweet honeypot post.  Being that Alaska doesn’t have many cities it would appear the person resides near Nome, Alaska.  This wonderful city of approximately 3,500 was featured in the Simpsons Movie and is a port for the southern point of the middle Alaska peninsula.  Even though small, Nome also features two airports.  Nome was first formed during a gold rush around 1890 and had the notable citizen of Wyatt Earp during its boom days.  The city had a problem with claim jumping so the US Government setup Fort Davis on the outskirts with 100 soldiers to keep the boom manageable.

Thank you Nome and welcome.

Powerful people make powerful foes with coercion, intimidation, and slander

“There is nothing concealed that will not be revealed, nothing hidden that will not be made known. Everything you have said in the dark will be heard in the daylight; what you have whispered in locked rooms will be proclaimed from the rooftops.” — Luke 12:2-3

“All that we are is the result of what we have thought: it is founded on our thoughts, it is made up of our thoughts. If a man speaks or acts with an evil thought, pain follows him, as the wheel follows the foot of the ox that draws the carriage… If a man speaks or acts with a pure thought, happiness follows him, like a shadow that never leaves him.” — Shakyamuni Buddha

“I submit that an individual who breaks a law that conscience tells him is unjust, and who willingly accepts the penalty of imprisonment in order to arouse the conscience of the community over its injustice, is in reality expressing the highest respect for the law.”  — Martin Luther King Jr.

“The very word ‘secrecy’ is repugnant in a free and open society; and we are as a people inherently and historically opposed to secret societies, to secret oaths, and to secret proceedings.”  — John F Kennedy

The facts are that leaks happen, have happened in the past, and will continue to happen.  People who leak that are motivated by immoral means do not just turn over their powerful and valuable information to the press and disseminate the information publicly.  They sell it, they trade it, and they keep it a secret.  Leaks to the press however are almost always painful yet enlightening for the public, serving a much greater good for the people while usually tarnishing the reputation of those which the information is about.  People who leak information are often called traitors, immoral, or are even threatened with their life. The current Wikileaks storm  is not the first time that questionable judgment by the government was brought to light.  Further, the alarmingly aggressive actions by the government at the highest levels is indicative of a much more sinister cover up by those in power.   In 1971 the case of New York Times Co. v. United States established that the freedom of press and information is above the government’s urge to censor and keep secrets.  The Pentagon Papers where the government claimed that the release would cause, “irreparable injury to the defense interests of the United States” turned out the be one of the catalyst that helped the American public stop their government from waging an endless war in Vietnam.  The Court decided that the freedom of press from censure is paramount to the freedom of the people.

The release of the Pentagon Papers derailed the government plan to expand the Vietnam war.  Also it showed that President Johnson was lying to the public and congress about the urgency of the threat of communism.  “We seek no wider war” he claimed running for President in 1964 yet continued to build the war not primarily for freedom or the Vietnamese people but only, according to a DOD internal memo, “to avoid a humiliating U.S. defeat”.  Putting our soldiers lives on the line and expanding the chaos and tragedy of war simply to avoid egg on the government’s face is truly the ultimate humiliation.

Around 1971, Daniel Ellsbery leaked the top secret documents because he saw the unjust war and the horror being unleashed by the government, lost and out of control.  He was harassed and called a traitor.  The president systematically organized a secret campaign called “The White House Plumbers” to smear Ellsbery’s name and used all their power to make anyone question their association or will to associate with Ellsbery.   This is the same committee which then went on to perform the cowardly Watergate burglaries leading to uncovering the Watergate scandal.  The government couldn’t ever prove that Ellsbery leaked the documents.

Fast forward to 2009.  Army Pvt. Bradley Manning, in Iraq since 2005, was well aware from his intelligence position of the brutal oppression and torture suffered by Iraqis from Americans in the name of preventing terrorism.  Pvt. Manning also had access to the secret plans and blunt accounts of top government official’s complacency and encouragement of perpetuating the horrendous acts all in the name of homeland security.  It is believed that Manning leaked the documents to Wikileaks.  Manning couldn’t keep his ego in check and bragged about the release of some recent leaks to Adrian Lamo.  Wikipedia references, “According to German newspaper Frankfurter Allgemeine Zeitung, a reporter from Forbes has published information, according to which Adrian Lamo works as a “security specialist” with “Project Vigilant”, a private security institution which also works for the FBI and the NSA”  Most likely the victim of indirect entrapment, Pvt. Manning is now facing up to 54 years in prison for removing the dark blindfold  placed over the American people’s eyes in the name of their security.

With the most recent releases, Secretary Clinton now says that the leaks are “an attack on the international community.”  This is the same rhetoric used by previous corrupt governments and their officials once they realized that their despicable and illegal actions were coming to light.  Even though news organizations all over the world are reporting and reading the leaks, Wikileaks and especially Jullian Assange is being singled out by what is by evidence of its own documents a brutal and unchecked government out of control in this war on terrorism.  The Denial of Service attacks against Wikileaks are not being condemned or even investigated.  Business associates are being coerced into disassociating themselves with the organization.  The government is showing no outrage for illegal attacks against websites and sits idly by as some representatives claim they would be comfortable with the murder of Assange.

The truth is that the United States government should never be involved in despicable acts against people in other countries or against its own people.  The freedom of the people to know and the press to keep us informed trumps the claim that nearly all secret information about government operations should be kept secret from you for your own protection.  Realize that truly the protection is not for you but the criminals which are operating in secrecy, wary of their transgressions being challenged and fighting tooth and nail to get back the powerful leverage they gain through secrecy.

The government fears the empowerment of an informed people and will always act to quell the the dispelling of this “secret” knowledge which gives them leverage against anyone else.  The cloak of secrets allow them to lie directly and have no repercussion of their actions, neither thoughtful discourse nor debate, only absolute power.  Their secret actions and collusion with powerful private parties are coming to light.  Do not have shame but relief that now the truth is revealed.  Let information of the government and its actions empower everyone, not just those who use these “secrets” for their own personal interests.

This post was inspired by the Institute for Public Accuracy release, “Ex-Intelligence Officers, Others See Plusses in WikiLeaks Disclosures

HB1443 is common sense

Stop wasting money on weed by telling your local Virginia General Assembly member to support HB1443:

Find your representative here:

Here is a sample letter you could send:

Delegate Toscano and Senator Deeds,

I urge you to support Delegate Morgan’s HB 1443 that is proposed for the upcoming session. As budgets are… squeezed, instead of laying off teachers or public safety officials, the state should quit taking an interest in subsidizing the persecution of private activities which some do not agree with. The facts are that non-violent marijuana offenders are the least of the state or law enforcements worries. Marijuana consumption is widely accepted as safer than alcohol or cigarettes yet it is much more regulated. Enforcing prejudice against a non-intrusive lifestyle choice is not in the government’s interest. It criminalizes otherwise law abiding citizens and breeds distrust where otherwise none is warranted. Substances abuse is a medical problem not a criminal problem. Help Virginia get on the right track by supporting HB 1443.