Code on Dan Goldin

Yahoo fantasy football stats: 2021-2022 edition

Sun, 29 Aug 2021 15:53:06 -0500

I had a bit of a blogging break this year but the pull of tradition is too strong. For the sixth year in a row I updated my Yahoo fantasy football stats scraper to pull Yahoo’s projected fantasy stats for the upcoming season. The only change this year was the addition of a “O Rank” column which was simple to implement so I have no excuse for pushing this off till the last minute. Hopefully this shames me into writing more.

Challenges in identifying a spawned OS X process

Tue, 29 Dec 2020 00:00:00 +0000

As part of my project to come up with a utility to deduplicate images on S3 I started working on a proof of concept to validate the flow and functionality. I know that the AWS S3 API has everything I need but I wanted to see whether I could use the built in OS tools to handle the image comparison without resorting to a webapp. The steps I had in mind were pretty simple:

Improving the accuracy of evergreen content

Tue, 29 Dec 2020 00:00:00 +0000

While looking through my blog’s analytics I noticed that I have a few posts that have views despite being written years ago. Some of these are still relevant while others are woefully out of date. For example, the difference between yyyy and YYYY in Java’s SimpleDateFormat is still relevant but how to scrape web pages has changed significantly since 2013 and anything that has pricing or performance information is likely wrong.

Removing duplicate files in S3

Fri, 18 Dec 2020 00:00:00 +0000

I’m a digital hoarder and whenever I had to switch computers, I was always worried about losing files. These days it’s both lower risk since so much is scattered across the cloud but with the ascent of AWS I’ve resorted to just backing up my computers onto S3.

I simply do a recursive copy of my home folder to S3 and call it a day. One problem this exposes is that there are duplicate files scattered all over the place. For example I’d have something both in my Downloads folder as well as in a Photos and maybe even a Dropbox folder. Or I would just have the same file duplicated in the same directory. At the end of the day it’s not a huge deal but at the same time it feels dirty so I started working on a script to identify these duplicates.

Amazon owns more than $2B worth of IPV4 addresses

Fri, 11 Dec 2020 00:00:00 +0000

While listening to a podcast discussing BGP I heard the fact that AWS owns more than $2B worth of IP addresses. I knew AWS was massive but this came as a big shock so I decided to do some digging around. I came across a site that listed the market prices of IP addresses and the range looks to be anywhere from $20 to $30 per IP depending on the block size. Now it was time to figure out the IP addresses owned by Amazon. I figured this would be difficult but lucky for us AWS actually publishes their entire set of IP addresses as JSON.

Bringing a user centric approach to the command line

Wed, 25 Nov 2020 00:00:00 +0000

I have a small script that takes an export of Apple Health data and then dumps it into a MySQL database that I then use to visualize my health metrics over time. My prior workflow was to open the app and then Airdrop the file over to my computer at which point I’d unzip it, copy the relevant file over into the script directory, and simply run the script.

Yahoo fantasy football stats: 2020-2021 edition

Tue, 18 Aug 2020 00:00:00 +0000

For the fifth year in a row I’ve updated my script to fetch the projected stats for the upcoming fantasy football season. These days I’m torn on football as a whole - both due to its politics and dangers - and don’t plan on watching too many games. Yet I enjoy the competition with my friends and the rote work of updating my scraping script to work every year.

This time around there haven’t been too many changes: Yahoo changed the order of a few columns and introduced some minor stylistic changes but the code only needed a few changes to work - much simpler than last year which required running with an adblocking extension to bypass a script blocker.

Coupling in action: protobuf enum fields

Mon, 17 Aug 2020 00:00:00 +0000

The next few blog posts are all topics that have come up recently in my 1-1s and serve as good examples for how I think about code and software engineering. The first post is a discussion around coupling and the tradeoffs we considered.

Our pipeline was covered in depth on the high scalability blog but the important part for this post is our event collection piece which consists of the following components:

Overthinking how to load data into MySQL

Tue, 30 Jun 2020 00:00:00 +0000

I have two projects that generate data to dump into my personal dashboard. One loads the health export from my Apple watch into MySQL and the other analyzes my blog posts and generates a CSV file of statistics that I then load into MySQL. The input to both is basically the same - either a file or a directory - and yet two different approaches to the processing.

Anatomy of a crypto mining hack

Fri, 29 May 2020 00:00:00 +0000

A few months ago I set up a simple ftp server to help a friend. I took a few security shortcuts which came to bite me this week when I received an alert from DigitalOcean that an instance was running hot.

I dug into it and noticed a series of processes being run by the ftp_user - the most impactful was a command called rsync. I’m familiar with rsync which syncs files across devices - and this was nothing like that. At this point I realized that I got hacked and quickly disabled the user and killed all its processes. The CPU usage dropped back to normal and now it was time to dig into the damage.

Metrics, logging, and error reporting

Wed, 27 May 2020 00:00:00 +0000

As software engineers it’s vital to understand how our applications are performing. The more information we have the better we can address problems, improve performance, and generally better solve problems for our customers. Imagine releasing a product with nothing being collected - you’d be flying blind.

I’m a huge advocate for measuring application performance in terms of business metrics and aligning it as much with the customer experience as possible. This post does not go into that. Instead, I want to share my beliefs around measuring the technical application performance. In my mind, they’re grouped into three major categories: metrics, logging, and error reporting. There is some overlap but I find it a helpful way to think about them.

Data analysis and visualization

Wed, 29 Apr 2020 00:00:00 +0000

Yesterday I had the rare chance to actually do some coding and realized how rusty I am at numerical analysis in Python. The task was simple - ingest a CSV that had a date column, two categorical columns, and a numerical column - and then generate a grid containing a series of line plots, each of which would be a combination of the two categorical columns.

I did a ton of this work years ago so knew what was possible. That’s half the battle and after a bit of searches I got a working solution. At the same time I’m disappointed it took me that long, especially after seeing the brevity of the end result. Software ate the world and now every business is generating tons of data. Being able to make sense of it is an increasingly important skill set especially if you’re a leader. It’s unfortunately an area I haven’t kept up with over the years and something I plan on remedying after this experience.

Optimizing code? Think theoretical limits

Sat, 18 Apr 2020 00:00:00 +0000

This post was inspired from a conversation with an engineer who was tasked with optimizing the performance of a heavily used static JavaScript script. This code gets loaded billions of times a day across a variety of devices and small improvements to its load time and performance can drive significant value to our customers and us.

When you start it’s easy to find the low hanging fruit and get the simple wins. But as you keep working on the same problem these wins become rarer and rarer and oftentimes a win in one area may be a loss in another. Using our static JavaScript code as an example we can argue that we should split it into multiple, smaller files and load them on demand. That would make the total amount of code loaded and executed lower but, on the other hand, would increase the number of network requests. And if you then layer in caching and the variety of devices it will run on it’s not at all obvious what the impact of a change will be.

Dumping Apple health data into MySQL

Sat, 11 Apr 2020 00:00:00 +0000

I apparently can’t get enough of Grafana and the latest quantified self push was to visualize the data from Apple health. Apple makes it pretty simple to export the data but it’s in XML so there’s a small bit of processing to turn into something that can be visualized. For my personal stats I’m dumping the data to MySQL and writing fairly simple queries to visualize them. Since I already did a similar export in my email-stats code I was able to reuse a fair amount. The major difference was that the Apple health export is fairly large (my export for 2020 was an 80 MB file) and it would be a shame to not apply a few optimizations.

Visualizing my journey to Inbox Zero

Tue, 31 Mar 2020 00:00:00 +0000

I subscribe to the “Inbox Zero” philosophy and treat my email inbox as a todo list that I slowly work through. As part of the desire to get more and more quantitative I wrote a quick script to pull the number of emails from my Inbox and then insert the data as a row into a new table in my personal stats database. As usual, most of the work was in deciding to do it and once I got to coding the hacky solution was done within 20 minutes. The script uses Python’s built-in imaplib library to log in to an email provider and then a simple MySQL query to insert the resulting data. I hooked this up to run every 15 minutes via cron and put together a Grafana dashboard to plot the count over time. I’m currently not actually going through the content of the email messages themselves but there are tons of directions I can take this - for example slicing the data by sender or examining the age of the messages. For now I’m just hopeful this motivates me to keep going through that email.

My personal Grafana dashboard

Thu, 27 Feb 2020 00:00:00 +0000

Last year I wrote about the idea of a personal dashboard and earlier this year I described my 2020 goals and how I’d go about measuring my progress. The past two days I was able to combine the two concepts and created a simple Grafana dashboard to measure my progress against the blogging goal. As with most tasks, the most difficult part was getting started and the actual exercise took a few hours. While it’s still fresh in my mind I want to document the step by step process in order to both provide a perspective into how I work while also giving others a guide to setting up their own.

Finding parking spots using YOLO

Fri, 31 Jan 2020 00:00:00 +0000

I finally managed to make a bit more progress on the Parking Spot Finder project. Last time I tested the computer vision products offered by Google and AWS to see how well they were able to detect cars. This time around I decided to actually start working on the computer vision side and found a nice tutorial that allowed me to quickly try out the YOLO computer vision library. Surprisingly, it did significantly better than AWS but was more mixed against Google. On the original image it did better than both AWS and Google at detecting the cars but it did worse than Google on the cropped images. The next step is to actually try training a model and see how well it can do. Depending on how that goes I may end up changing my approach up and see how far I can get using heuristics.

Parking spot finder

Thu, 26 Dec 2019 00:00:00 +0000

I live in Jersey City and rely on street parking rather than paying for a lot. Jersey City also some aggressive alternate side parking rules with many streets having twice-a-week cleanings which in the worst case requires me to move my car 4 times a week.

A few years ago I decided to build a simple tool that would be hooked up to a camera and would notify me whenever a spot became available. I unfortunately never finished that up and am hoping that by posting it publicly it will act as a motivator and hold me accountable.

SMS based password manager

Tue, 19 Nov 2019 00:00:00 +0000

Two Fridays ago I wrote about my home-built password manager and while working well on a computer it was a pain to use on mobile. I’ve been trying to think of a more friendly way of supporting this on mobile and came up with the idea of retrieving the passwords via SMS. The workflow would be to send a text to a number with a search term and then get the password back as a response. Twilio made it surprisingly easy to build a simple proof of concept (GitHub code) and the resulting code is fewer than 50 lines and supports the use case.

DYI password manager

Fri, 08 Nov 2019 00:00:00 +0000

I prefer plaintext for all my note taking. Text files are extremely portable and serve as a flexible foundation for anything I’d want to do. For example, I can use grep to perform complex regex searches across thousands of files. If I want to apply a bulk operation to my notes I can write a quick script to do so. This is all possible because there’s no proprietary format backing them and I’m able to leverage the power of the command line.

Upgrade your libraries

Mon, 04 Nov 2019 00:00:00 +0000

This blog is hosted on GitHub and built using Jekyll. Jekyll is a simple static site generator that’s been working well for me and was flexible enough to allow me to switch the entire site over to AMP. Unfortunately, the switch to AMP led to the site generation becoming significant slower due to the CSS-inlining requirement. Two years ago I started profiling and dropped the site generation time down to about 15 seconds from over 4 minutes by generating the CSS once and then reusing it for all future pages.

Deplying Docker using systemd

Mon, 28 Oct 2019 00:00:00 +0000

A few months ago in a burst of inspiration I converted a bunch of my side projects to run inside Docker. Unfortunately, I didn’t do the follow up work of actually creating a Kubernetes cluster and instead came up with a ridiculously hacky process to get them running. I created a simple shell script that would just build and run a Docker image and have just been running it inside a screen session.

File size histogram via the command line

Sun, 20 Oct 2019 00:00:00 +0000

I’m a sucker for a good shell command and recently discovered (via StackOverflow) the most complex one yet - a one liner to generate a histogram of file sizes within a directory. The sizes are in powers of two but it’s a great way to get some simple summary statistics of files inside a directory. I still find awk mystifying to write but nearly every advanced shell command uses awk in some way. Most engineers thee days have a bias for a traditional scripting language but it’s still amazing what an awk one-liner can do.

Visualizing my Twitter archive - 2019 edition

Mon, 07 Oct 2019 00:00:00 +0000

GitHub sent me an alert this past weekend that a bunch of my repos were using old libraries that had security vulnerabilities. Nearly all of them were due to my usage of an old version of the requests library. Updating those was as simple as updating the requirements.txt file to the new version.

One of these repos, twitter-archive-analysis, is my most popular project on GitHub so I thought I might as well revisit it and see if I could both address the vulnerabilities and get it running again. Upgrading the packages was straightforward but there are very few things more humbling than looking at the code you’ve written years ago. Twitter changed the format of the archive from JSON to CSV since the last time I ran the code and as part of the upgrade I did a little bit of cleanup. The code is up on GitHub and I’ve included the visualizations it generated below highlighting my Twitter behavior over the years.

Upgrading pip packages within a Dockerfile

Thu, 03 Oct 2019 00:00:00 +0000

I have an old project, makersalley.com, that used to run on an old version of Python (2.7) and an archaic version of Django (1.4). Earlier this year I overhauled it to run on a newer version of Django (1.11) and Dockerized the entire setup which required all sorts of changes and library fixes.

Last night, I took it one step further by upgrading it to the latest versions of both Python (3.7) and Django (2.2). The most difficult part was figuring out how to upgrade to the latest versions while being tied down to Docker. For example, changing the Python version in the Dockerfile caused many of the packages in requirements.txt to not build but because they were all executed within Docker I had to get the entire requirements.txt fixed before that step would succeed. Similarly, it wasn’t clear which versions of the packages in the requirements.txt depended on one another and upgrading all of them blindly would have been a fool’s errand. It also turned out that one package, MySQL-python, was not available in Python 3+.

Visualizing Kafka partition changes

Tue, 01 Oct 2019 00:00:00 +0000

Earlier this week we scaled up our Kafka cluster to take advantage of more availability zones and increase the replication for some of our key topics. After making sure the new brokers joined the existing cluster we needed to redo the partitioning to take advantage of these newly available brokers.

I’m sure there are better and more modern tools out there but we’ve been using SiftScience’s kafka-assigner. Rather than being a naive partitioning it looks at the existing assignments and optimizes the new assignment to minimize the number of moves while striving to keep the partitions evenly distributed across all brokers.

Solving Num: A combinatoric math game

Sat, 14 Sep 2019 00:00:00 +0000

I don’t play games on my phone but one game I keep going back to is “Num” - an “insanely hard math game.” The premise is pretty simple - you have a few numbers that you need to combine, using the four basic math operations, so it computes to a specific number.

The levels start off simple but it gets more difficult with a lot of trial and error at the higher levels. They’re no longer simple expressions that can be evaluated fairly linearly but instead are solved by using fairly complex intermediate values.

In praise of tcpdump

Fri, 13 Sep 2019 00:00:00 +0000

We switched over to Prometheus and Grafana for our monitoring but some of our older systems are still on Graphite and StatsD. One of these is an alert for disk usage that started going off a few weeks ago. Over the course of the day it kept fluctuating from 100% disk usage to ~40% and whenever we dug into it we only saw the 40% number. Since StatsD is push based we assumed it was another instance that was submitting its metrics under the same key. Unfortunately, because StatsD is push based, it wasn’t clear which instance was actually doing the conflicting metrics push.

Explain the why in code reviews

Sun, 25 Aug 2019 00:00:00 +0000

This may be obvious to most people but it’s still worth reiterating: if you’re leaving a comment on a code review make sure to explain your reasoning. Code reviews are a key component in writing high quality code, improving everyone’s skills and knowledge, and encouraging a strong and collaborative team.

Code reviews can be a bit clinical with feedback being blunt and a few words and it’s important to add context to all but the simplest of feedback. Experiences and backgrounds vary across the team and what’s obvious to one person may not be obvious to others. Similarly, it softens the feedback and encourages both sides to have a conversation. The act of describing the why also forces the reviewer to slow down and actually think about their reason for the change request. This may, in fact, reveal that the change request is not actually useful or may not actually have a good reason.

Yahoo fantasy football stats: 2019-2020 edition

Mon, 05 Aug 2019 00:00:00 +0000

In what has become an annual tradition I’ve updated my Yahoo Fantasy Football scraping script for the 2019-2020 NFL season. The script works by logging into the Yahoo Fantasy Football site and downloading the first 4 pages of projected stats for each week. The code is up on GitHub as well as the stats in a CSV file.

Every year there’s been something that Yahoo did to break my script and this year was no different. The first thing I discovered was the login broke despite the HTML elements staying the same - it turns out my login “click” event was actually clicking on an ad which was opening up a new window. Changing the code from a click to an enter solved that problem.

Keeping Ubuntu computers in sync

Sat, 13 Jul 2019 00:00:00 +0000

I switched to Ubuntu a few years ago and along with the change decided to keep as much of my environment setup in version control as possible. The motivation was to make the process of setting up a new computer as simple and repeatable as possible since I planned on shifting both my personal and work computers to Ubuntu and wanted to keep them in sync as much as possible.

Video of my Data Council NYC talk

Sun, 07 Jul 2019 00:00:00 +0000

Last November I gave a talk at Data Council NYC ‘18 titled “The Highs and Lows of Building an Adtech Data Pipeline” and finally saw that the video has been uploaded to YouTube. If you’re interested in hearing a runthrough of the different iterations our data pipeline went through over the course of 6 years definitely give it a watch and leave comments and feedback. I’m not the most natural of public speakers and there were moments I spoke much quicker than I should have but hopefully having the slides on the side make it a tad more understandable.

Tightly coupled data loss

Sat, 15 Jun 2019 00:00:00 +0000

A few months while doing some Kafka maintenance we ran into an issue that caused us to lose approximately 10% of records across all our topics over the course of an hour. It was a big screw up but what made it worse was the interaction between the records. Our records represent the life cycle of an ad from the auction, to the render, to whether it was in view, and beyond. Rather than send all the information along for each on we keep the records as lightweight as possible and rely on our backend processing to join them together to come up with an auction log.

Modernizing Makers Alley and Better404

Sun, 27 Jan 2019 00:00:00 +0000

Years ago I started two companies, Makers Alley and Better404, and while they were both failures I didn’t have the heart to shut down the site. I put a lot of heart into building them and just pulling the plug felt cold. Instead I just paid the annual hosting fee and had them both running on a small AWS instance. This worked fine for years but unfortunately while I was messing around with Terraform I ended up terminating the instance entirely. Beyond the fact that they were both running incredibly old libraries with no available pip libraries the instance also contained the only copy of the database. I wa stuck with having the source code from a few years ago and not much else. Rather than give up I decided to use this as an opportunity to learn a bit and modernize them using some of the newer technologies.

Secret management across computers

Wed, 26 Dec 2018 00:00:00 +0000

As part of the Cyber Monday nonsense I convinced myself to purchase a ThinkPad in order to run Linux. I have a separate post coming about the transition from OS X to Ubuntu but one area I’m still trying to get under control is secret management. I have a ton of code on the old computer and almost all of it is on GitHub. The biggest challenge so far has been migrating secrets across computers. For example I have a variety of scripts that talk to Slack or the Google ecosystem which require their own tokens and keys. Some are straightforward to reissue - for example AWS - while others require a fair amount of frustrated searching. I don’t have the answer here but have been thinking about the following approaches to make things easier and would love to know how others handle this.

New code is not a linear increase in complexity

Sat, 15 Dec 2018 00:00:00 +0000

When it comes to software development we often add features simply because it’s easy. And almost always they are - just add an additional optional argument or two to a function and suddenly you’ve expanded your application’s functionality. The catch is that this assumes that this new code is a linear increase in complexity but it’s not.

Computer science has the concept of “Big O notation” to measure how a function behaves as a function of it’s input. A “Big O” of O(N) is linear while O(N^2) is quadratic. The implicit goal is that you should strive to write your code to minimize it’s complexity with the ultimate goal being O(1). The same approach can be applied to code complexity. How much will the new functionality affect the complexity of the code? A seemingly simple code change may change the “exponent” of your code’s complexity and a complex code change may actually reduce your code complexity. The code change is not always correlated with the complexity that is being introduced.

Counting the number of lines of code in a GitHub account

Thu, 13 Dec 2018 00:00:00 +0000

It’s surprisingly difficult to count the number of lines of code in a GitHub account. One day I’d like to come up with a fully automated solution but in the meantime I’ve come up with a workaround that gets me what I need.

Follow the steps in the following Stack Overflow answer to create your own command, cloc-git, that fetches a repo and runs another utility, cloc, that counts the number of lines in a git repo.
Get all your repos into a single file, one per line.
Bulk edit the file to have each line be an invocation of the clock-git command and save them all to a single file. For example, a single line of the file should be of the format: cloc-git git@github.com:dangoldin/dangoldin-blog.git
In the command line simply execute the file and pipe into an output file, for example sh loc.sh > lines-of-code
Once the previous step succeeds you’ll have a single text file with the output of the cloc-git command for every specified repo but the formatting is not the easiest to follow.
Run a simple grep command to get every line containing the SUM line: grep “SUM” lines-of-code and save this to the clipboard
Unfortunately the spacing is all off so you can’t use the cut command to do a split via the shell so you have to use a simple programming language. I used python and just dumped the contents into a single variable and ran the following command to split it into lines and then retrieve the last value when splitting by a space.
Once you have these values just do a simple sum to get the total number of lines.

It’s not very simple and forces you to use a variety of tools to get to the final result, ranging from reading Stack Overflow documentation to some shell commands to some Python scripting. It’s a good example of where having enough breadth of knowledge and experience with a variety of tools turns a hairy problem into one that can be solved relatively quickly. It’s not a perfect solution but for a one-off I’m happy with the results.

Aggressive code deprecation

Sun, 25 Nov 2018 00:00:00 +0000

Part of writing high quality code quickly is deprecating no longer used features and functionality. It sounds simple but more often than not there’s an abundance of references throughout - some tightly coupled and others loosely coupled - that make a full deprecation difficult. In some cases it’s is as simple as an isolated code change while in other cases it’s removing code along with some database migrations and in the extreme case it may be removing an entire service. It’s crucial to be exhaustive in your deprecation or you’ll end up in a situation months or years later where the team has changed enough that no one can tell what the code is meant to do and whether it’s still used.

Code without online help

Sat, 24 Nov 2018 00:00:00 +0000

Whenever I need some coding help my first step is to do an online search which usually leads me to either the library documentation or a StackOverflow page. This is a poor habit and something I’m trying to move away from. While I’m almost always online it’s dangerous to rely on the internet to code - both because there will be times you may not have internet access but also because you lose the ability to do your own investigation, discovery, and critical thinking.

Python 3 and aiohttp

Fri, 16 Nov 2018 00:00:00 +0000

A few months back I read about aiohttp and asyncio and finally got the chance to play around with it a few weeks back. The project was a quick one-off scrape of a few thousand domains to see what percentage had implemented the pubvendors.json spec, an extension of GDPR that allows publishers to specify the vendors they’re working with.

My initial reaction was to do it in the way I’ve done it countless times before: the requests library in a for loop. Instead I decided to actually try something new and use the aiohttp library, a new asynchronous library for Python 3. It took me a little bit of time to figure out how to structure the code and use Python’s new async functionality (which by the way is very similar to modern JavaScript) but the end result is simple for what it does and runs incredibly quickly.

Yahoo fantasy football stats: 2018-2019 edition

Sat, 18 Aug 2018 00:00:00 +0000

This is much later than in previous years but hopefully that just makes the data more accurate. I updated my Yahoo fantasy football stats scraper to account for the slightly different design for the upcoming season. It still works as before and uses Selenium to open up Chrome and scrape the projected stats by week. The change this year involved shifting the columns around a tiny bit as Yahoo changed the order but other than that there were no changes. Maybe by next year I’ll update the script to be able to actually determine the column indices for each stat automatically. As usual, the script is up on GitHub and the scraped data can just be downloaded here.

MySQL foreign keys

Sat, 07 Jul 2018 00:00:00 +0000

Databases are the last layer of defense against corrupt data and the more restrictive you can make them the better. No matter how much validation you may have missed in your code having a strong and restrictive database schema will protect your data. One of the best approaches to building a restrictive schema is using foreign keys which specify how fields from one table relate to the fields of another table. There are a few options here and make it possible for you to specify anything from automatically removing rows when a row they’re referencing is removed to recursively updating rows when their references have changed.

ALB and ELB access log schemas for Redshift

Tue, 05 Jun 2018 00:00:00 +0000

Back in February I wrote about using Redshift to quickly analyze ELB access logs. This worked great until we switched from using ELBs to using ALBs. Unsurprisingly in hindsight but frustrating at the time the ALBs have a different log schema. Both the Classic and Application Load Balancer logs are well documented on the AWS site but unfortunately the code to create the appropriate Redshift schema is not. In the hope of helping others and passing it forward I wanted to share the Redshift schemas for both types of access logs.

Power of shell commands

Sat, 26 May 2018 00:00:00 +0000

It’s surprising how unappreciated shell commands are. They’re incredibly powerful and once understood are able to handle small one-off tasks much quicker than writing even simple scripts. Earlier this week I ran into a small task that highlights the power and ability of the shell.

A few of our applications use the same configuration file which contains a variety of URLs, secrets, and passwords. If any of these applications require a field the it gets added to this growing configuration file. This is clearly not good for security and as part of a larger security revamp we’re moving to application-specific config files. Long term we want to revamp the way we do deploys such that the configuration is kept in environment variables and handled by the build system but as a short term solution we want to split this single configuration file into a file per application.

Connect Four bot competition

Wed, 25 Apr 2018 00:00:00 +0000

Years ago when I worked at Yodle the engineering team held a Connect Four bot competition. The goal was for each person to write a Connect Four playing bot and then let them loose to determine the winner. We had either a few days or a few weeks to do this and my failed approach was to use genetic programming to evolve a bot. The best it did was beat a completely random bot 80% of the time while the winning entry leveraged Minimax with Alpha Beta Pruning.

Analyzing AWS ELB logs

Tue, 20 Feb 2018 00:00:00 +0000

Logging HTTP requests should be enabled for every application you run. When things go wrong, and they will, it’s often the first step to understand the problem. Unfortunately, logging isn’t always top of mind and is often forgotten. Luckily, if you use the Elastic Load Balancer (ELB) functionality within AWS you’re able to set up ELB logs that track every request and write it to an S3 bucket. The documentation is up on the Amazon site but there’s a surprising amount of information that’s hidden away in the logs. Since it’s Amazon and they want to make it as easy for you to use their various services together it’s simple to load the logs into Redshift and start digging into them via some basic queries.

Phonetic distance

Tue, 16 Jan 2018 00:00:00 +0000

Last year I wrote a simple script to automate posting our On-Call schedule. It worked by reading the schedule from a Google Spreadsheet, looking up the names in Slack, and then sharing these usernames on Slack. A tiny problem I ran into was the fact that since I was using an exact match the names in the spreadsheet had to match the names in Slack. This is a trivial problem to solve since we have a finite number of engineers but it still felt a bit too sensitive. While lying in bed last night I got to thinking of ways to measure similarity between the names in order to make it a bit more fuzzy. I’ve used the Levenshtein distance in the past but it felt a bit too clinical for what I was trying to do and I wondered whether it was possible to do a phonetic match.

Learning Docker

Wed, 03 Jan 2018 00:00:00 +0000

I’m a bit embarrassed to admit this but I’ve been a bit behind the Docker craze. Sure I’ve done the tutorials when it came out but never really applied it to any of my actual projects. Given that nearly everyone is using Docker in some shape or form, I decided it was finally time to give it an honest effort.

I had a small Python script that I’ve been running weekly off of my laptop and wanted to come up with a better solution. My old approach would have been to just run it as a cronjob on a VPS but the problem was that it had a variety of third party libraries that needed to be installed via pip and if I ever needed to move it elsewhere I’d have to set it up again. After speaking with a few people it seemed that this could be a use case for Docker and then running it either within a build server (Jenkins) or on top of a distributed system (DC/OS).

Analyzing my blog: 2017 edition

Thu, 21 Dec 2017 00:00:00 +0000

I have a set of scripts I wrote in 2016 that aimed to analyze my posts over the years and hopefully offered up some insights. I’ve updated them for 2017 but rather than posting every single visualization I thought it would be more valuable to highlight the ones that seemed the most relevant and interesting.

The year is not quite over but I'm defintiely behind on my posts that I hope to power through by the end of the year.

Jira analysis script

Thu, 07 Dec 2017 00:00:00 +0000

A few days ago I wrote about using average number of sprints to complete a story as a way to measure a team’s sprint efficiency. Unfortunately at that time I had a pretty hacky Jira analysis script that I was too ashamed to share but it has been cleaned up enough for me to not feel too much guilt. It’s available on GitHub and comes with a few additional bells and whistles. One is specific to the way we work where we label relevant stories with a priority (priority:1, priority:2, etc) based on our planning process so we can hold ourselves accountable to spending the appropriate time on our initiatives. The other is a simple way that calculates the story points done by assignee. It’s a dangerous metric to use since story points are variable and not all work is measured via story points but it’s yet another metric that can help highlight or sharpen a potential issue.

Improving Jekyll generation speed for AMP pages

Thu, 23 Nov 2017 00:00:00 +0000

Last September I migrated my blog over to AMP which entailed a variety of challenges ranging from converting every img tag to an amp-img tag with some additional metadata to figuring out how to support Disqus. I tackled the critical ones but the one I never got to was speeding up the build time since it had no impact on the actual reader experience and just slowed down my build and commit process. During this Thanksgiving break I finally decided to do something about it after discovering that jekyll has a profiling feature. It’s expected that the bulk of the time is spent generating the post pages but running the profiler highlighted that the majority of the work wasn’t in the actual content block but in generating the head element - something that shold similar from page to page.

Archiving large MySQL tables

Sat, 18 Nov 2017 00:00:00 +0000

One of the major changes we made when building the latest iteration of our data pipeline was moving our key agg tables over from MySQL to Redshift. Despite the migration we thought it would be prudent to archive these tables. The challenge was that some of these tables were hundreds of gigabytes so doing a simple mysqldump wouldn’t work. The reason these tables were so large is because they included a date dimension which led to our archive script. The script works by generating a sequence of shell commands that slice the table into chunks by date, gzip each chunk, and upload it to an S3 bucket. This keeps each individual chunk small enough to archive while making sure all the data is captured. It’s not the most elegant solution but it’s obvious and it’s quick. The one piece that’s missing is the table schema which can be fetched separately.

Spark's read.jdbc

Tue, 07 Nov 2017 00:00:00 +0000

Yesterday I spent a bit of time investigating one of our Spark jobs that had suddenly shot up in run time. The purpose of our job is to collect all the events we see in an hour and generate a variety of aggregate tables and files that can then be loaded into various systems. When we first wrote the job it took about 45 minutes to run but as we’ve started seeing much higher data volume the job time has crept up to to 90 minutes. And for some reason yesterday the jobs were not completing even after 2 hours. There was clearly something odd happening.

Philosophy of code

Sun, 29 Oct 2017 00:00:00 +0000

After writing my post on the code review pyramid I realized that I had many more thoughts about the highest level, code philosophy, and wanted to dedicate a full post to dig into it. The general idea is that a highly functional engineering team is way past the point of arguing over style and syntax and has reached the point where they share the same code philosophy. At this point all members of the team have an instinctive sense of how and where new code should be written - even if they can’t necessarily explain it.

The code review pyramid

Fri, 27 Oct 2017 00:00:00 +0000

Every modern software development process contains some form of code reviews. They ensure that all code is looked at by someone other than the author. This improves context, increases code quality, and generally leads to a stronger team and product. Yet there’s a world of difference in code reviews and I I’ve started to think of the different types as a pyramid. The peak is at the highest level but it’s not possible to get to that without going through the lower tiers. Another observation is that the lower levels can done by an individual but the penultimate ones require a team effort.

Schedule automation using Google spreadsheets and Slack

Fri, 20 Oct 2017 00:00:00 +0000

Back in March I wrote a script that would go through an on-call calendar kept in a Google spreadsheet and then post the current week’s schedule to a Slack channel. This worked surprisingly well and I thought of doing something similar for the other engineering team calendars. In addition to the on call rotation, we have a dedicated time for internal tech talks as well as a session to cover the news in the industry. To make them easier to manage we keep them all in that same spreadsheet.

Downloading your AIM buddy list

Mon, 09 Oct 2017 00:00:00 +0000

While writing the most recent post about the impending AIM shut down I became curious and logged in to see what what I’ve been missing. The application felt worse but seeing my buddy list made me nostalgic and going through the usernames brought back some fond memories as I tried to remember who each screen name belonged to.

I’m a bit of a hoarded, across both the physical and digital worlds, so didn’t want to lose my buddy list after the shut down. Unfortunately, AIM doesn’t make it very easy to download a copy of your buddy list so I came up with a crude but effective approach. Normally I’d look at the source code, identify the HTML elements containing what I wanted, and write a little bit of code in the JavaScript console to extract what I needed. I tried this approach in the AIM web client but it turns out that they update the HTML code to only show the screen names that are actually in view and it wasn’t obvious where the full list of screen names was being stored.

Generating a series of commands covering a date range

Thu, 05 Oct 2017 00:00:00 +0000

I know the title of the post is terrible but I found it difficult to describe the content in another way.

Lately I’ve been spending a decent amount of my time in SQL-land and running some pretty repetitive queries where only some of the arguments are changed. These run the gamut from exporting some data for a date range by day to adding a series of date partitions while messing around with Spectrum. Depending on the amount of these queries I needed to write I’d either just do it manually with a bunch of copy and pastes or use Excel to generate the queries I needed.

Examining my shell command history

Thu, 21 Sep 2017 00:00:00 +0000

A few years ago I wrote a simple script to analyze my shell history in order to examine my most frequently run shell commands. Being in dire need of a new blog post and suffering from a pretty heavy bout of writer’s block I thought it would be interesting to rerun the analysis and see how it compared to results from over 3 years ago.

It’s tough to say whether my usage has changed significantly. My adoption of zsh with the oh-my-zsh plugin have made my usage a bit more efficient - especially when using git. The other obvious change is that I’m running python code half as much as I used to and have also reduced my usage of text editors. I used to use fabric a ton to automate some deploys but have moved completely off of that. What has remained consistent is my blogging - my little alias (cdblog) to move to my blog directory and jekyll have stayed roughly the same as well as some other administrative commands.

A unified Lambda architecture

Fri, 25 Aug 2017 00:00:00 +0000

Lately I’ve been thinking about the Lambda architecture used in modern data pipelines. Lambda architectures are designed for systems that contain massive amounts of streaming data that needs to be processed and exposed quickly. The architecture consists of two different systems. One is a real time pipeline that’s not perfectly accurate but is able to handle large volumes while providing a solid estimate quickly. The other is a batch process that is accurate but runs on a delay. By combining the two you get the best of both worlds - accurate historical data and reasonably correct recent data that will be corrected by the batch job when it runs.

JSON to CSV

Sun, 20 Aug 2017 00:00:00 +0000

A while back I needed to dump some some EC2 instance information into a CSV file for a quick analysis. Just to get it done I took the immediate approach of using the AWS API to pull the details and then just navigating the massively deep structure. This approach required code designed for that exact structure so it got me thinking of a more generic approach that would be able to extract CSV data from an arbitrary JSON structure. It’s a surprisingly tricky problem since JSON consists of both lists and dictionaries and can have a pretty hairy nesting structure. Just to get the EC2 instances one has to go through a list of reservations each containing a list of instances with the various fields at different hierarchy levels - and some depending on another value within the same structure.

Visualizing my meetings over time

Fri, 28 Jul 2017 00:00:00 +0000

As part of never ending goal to improve my efficiency I was curious to understand how my meeting habits have evolved over time. I had an old script that would identify meeting room hogs and repurposed it to just download every one of my calendar events from when I joined TripleLift and another small script to analyze this data. Two things I had to filter out were multi day events which were tended to be vacations and events with me as the only attendee which were my reminders and todos. Unsurprisingly, there was a pretty large increase over time as we grew from a scrappy startup of 15 people to one with over 150 and as my role evolved from an individual contributor to a manager and then to the head of the engineering team.

Yahoo fantasy football stats: 2017-2018 edition

Sat, 08 Jul 2017 00:00:00 +0000

In what has become an annual tradition I updated my Yahoo Fantasy Football scraping bot for the 2017-2018 season. Every year Yahoo makes a few changes to their page and this year was no different. It’s always fun to cross my fingers, run the script, and see what breaks. This year the changes were surprisingly minor. For some reason Yahoo changed the name attribute of the password field from “passwd” to “password” and made a few tweaks to the table structure which required updating the XPath selectors. Other than that everything worked as expected and the 2017-2018 data is available here with the code up on GitHub.

Thoughtful code

Tue, 04 Jul 2017 00:00:00 +0000

Lately I’ve found myself thinking more deeply about the code I’m writing. No matter how small the task or script I’ll think through the implications of my approach and whether I should be doing anything differently. This doesn’t mean I’ll always pick the more correct and flexible approach and more often than not I’ll choose the quick and dirty one to save time but the thought process itself is valuable since it gets me in the habit of questioning and constantly improving my code. The following is an example that illustrates this approach.

Send private messages to all members of a Slack channel

Fri, 30 Jun 2017 00:00:00 +0000

One of my more recent “management automation” tricks was to write a simple script that gets all active members of a Slack channel and then sends them a direct message. I’ll often want to poll the entire team and ask them to fill out a survey or submit a questionnaire but the response rates tend to be poor. But if I send a message to people directly I end up with a much better response rate. It turns out that in my case I was able to get a greater than 100% improvement in response rate by using this approach. In a group channel there’s a lot going on so it’s likely that some people don’t see the message or decide they’ll do it later but inevitably forget. But by messaging them directly it sends a pretty strong signal that I care about the response and prompts people to just get it done. Despite the success I am hesitant to overuse it since it may lead to people ignoring these direct messages as well. As they say, with great power comes great responsibility. As usual, the code is up on GitHub and suggestions and pull requests are welcome.

Getting AMP into RSS

Tue, 20 Jun 2017 00:00:00 +0000

A little less than a year ago I migrated this blog over to AMP which required a lot of small tweaks - ranging from automating the markup changes to getting the Disqus plugin to work. One thing I didn’t get a chance to finish until earlier this week was supporting the RSS feed. This blog is hosted on GitHub pages which is powered by Jekyll and comes with a pretty powerful templating engine. One of the predefined templates was the ability to generate an RSS atom feed. It worked by taking the content of each post, escaping it, and concatenating them together into a massive XML file.

Copying production SQL data to other environments

Sat, 20 May 2017 00:00:00 +0000

I suspect most developers have encountered this problem at least once: how do I copy some production data to my test or development environment? This can stem from needing to fix a bug that only manifests in production or just getting a more complicated, real-world dataset that doesn’t yet exist in the test environment. In an ideal world we’d have everything we need in fixtures and properly tested but in the cases we don’t it seems simpler to just copy the data over from the production environment.

Security across multiple AWS regions

Thu, 04 May 2017 00:00:00 +0000

As great as AWS is there’s still a major gap in the way cross-region support are handled. It’s boggling that there’s no single screen to see every one of your instances and you’re forced to do it a region at a time. Beyond the cosmetic it’s not-obvious how to get instances from multiple regions to communicate securely with one another. On one hand Amazon has the neat concept of a Virtual Private Cloud (VPC) that allows you to create a group of machines that act as if they’re on the same network. This makes it simple come up with some pretty neat security rules - for example only allowing for an instance to communicate with the outside world via port 80 but with its network on other ports. Using a combination of VPCs and security groups one can come up with a pretty intricate security system.

Having some fun with the RGB color model

Sun, 30 Apr 2017 00:00:00 +0000

The best way to learn a new technology is to play with it so to learn React I started a simple project I termed “color-fun" (GitHub). The general idea is to let you specify a starting color along with a step size for each of the digital primary colors and see the color progression. By messing around with various combinations one can get a pretty good sense of the way the RGB color scheme works. To make it a bit less boring there’s also a “random” option to generate a new value combination and a new color row.

Refactor driven development

Sun, 19 Mar 2017 00:00:00 +0000

There are a variety of software development methodologies and I’d like to throw another one into the fray - refactor driven development. Rather than focusing on tests or models or functions the focus should be on expressive and maintainable code. Imagine spending 80% of your development time on refactoring old code and laying a solid foundation for all future work. Then the remaining 20% of the time can be spent on writing new features and functionality that drive the product forward. Once this work is done it may lead to more refactoring work to get the code back into a pristine state.

In praise of long running code

Sun, 12 Mar 2017 00:00:00 +0000

There’s something spectacular in checking in on a project you worked on years ago and discovering it’s still running years later. This past Friday I got an HTTPS alert from Let’s Encrypt reminding me that my SSL certificate for https://yahnr.dangoldin.com was set to expire. I checked it out and remarkably it’s still up and running. I built that in March of 2013 as a proof of concept of what I termed a “pseudo-static site.” The idea was to have a site hosted on statically on S3 but powered by a dynamic job that refreshes the underlying data.

Automating admin work: Spreadsheets to Slack

Sat, 04 Mar 2017 00:00:00 +0000

Recently we adopted the concept of owning your own up time for our engineering teams. The goal is to encourage a stronger sense of ownership and actually give the teams the autonomy to approach their development and release process the way they’re comfortable with. Before this we relied on a single on call every week that would be responsible for monitoring all issues and escalating them to the appropriate team. One minor side effect of this change was that I now had to manage the on call calendar and post the new rotation on Slack every week. Since this was a good opportunity to mess around with the Google Spreadsheet and Slack APIs I decided it good be a fun little project.

Lessons learned from today's S3 failure

Tue, 28 Feb 2017 00:00:00 +0000

Today was quite a day. S3, the most resilient of Amazon’s services went down for a few hours in the US-EAST-1 zone and led to a series of failures across a variety of services. There are a ton of lessons one should take away from this - ranging from running across multiple availability zones to being integrated with a variety of cloud providers. The challenge is that it’s not easy; especially when you’re small. At that point you have to prioritize building support for a 0.01% chance of massive failure versus a variety of features and product enhancements to drive your business forward.

Learning modern frontend development

Wed, 01 Feb 2017 00:00:00 +0000

Until a few weeks ago my frontend programming experience ended with jQuery so I decided to do something about it and start getting up to speed with modern frontend development. This ranged from starting to mess around with React, to using ES6, to integrating webpack and Babel in these projects. I’ve been using Sublime Text for the past 6 years but am switching to Visual Code Studio as my primary editor.

Year in review: 2016

Mon, 02 Jan 2017 00:00:00 +0000

A hallmark of blogging is to do a year in review post with every blogger having their own distinct style. Some write about their tops posts, others about the lessons learned, some focus on the books read or places seen. I’ve been keeping meticulous daily stats around the hours slept, my physical and mental states over the course of a day, as well as the food, coffee, tea, soda, and alcohol consumed and the review is an opportunity for me to summarize and visualize this data. The goal is to identify healthy and unhealthy trends over time and use that information to make changes in my life. At the moment the stats are mostly high level summaries but what I want to do is use this data in order to identify hidden relationships in order to improve my physical state and mental moods. This is a work in progress but I hope to do more of that this coming year as well as improve the way I’m gathering this data. The analysis code is up on GitHub with a guide and a sample file that can be analyzed. And now on to the data:

Automatically taking screenshots of HTML elements

Tue, 13 Dec 2016 00:00:00 +0000

I’ve worked on a variety of scraping projects that required spinning up a browser (via selenium) and having it browse a variety of pages unattended in order to capture some data. The two most recents ones scraping my account data from Turo and the fantasy football stats from Yahoo. These were relatively straightforward since the browser was used purely to navigate from page to page with the actual data capture done by parsing the underlying HTML.

Word clouds and text similarity

Sat, 10 Dec 2016 00:00:00 +0000

I’m a sucker for data visualizations so when I came across a simple word cloud-generating Python script I knew I had to give it a shot. Lucky for me I’ve been blogging fairly consistently since the beginning of 2013 and have a large text set to visualize. The first step was generating a word cloud for every single post I wrote and the second was to break it down by year. This didn’t reveal too much but got me thinking about how my writing has changed over the years. This led my discovery of a script on StackOverflow that works by translating each block of text into an tf-idf (term frequency - inverse document frequency) vector and then calculating the cosine distance between them. This intuitively makes sense. The tf-idf vector is used to highlight and quantify the unique words in a given document as a vector and the cosine distance is used to compare the similarities between them - if they vectors are equivalent the angle between them is 0 which has a cosine of 1. Turns out that high school math is incredibly useful.

Efficiency vs expressiveness

Tue, 06 Dec 2016 00:00:00 +0000

The ideal code is both efficient and expressive but they’re often at odds with one another. Last week I was working on a simple script to parse and visualize a detailed AWS bill across a variety of dimensions and came across a clear example. The script loads a CSV file into a Pandas dataframe and adds a few columns based on the values of some others. The challenge is that the CSV file can be millions of rows so minor improvements can lead to significant efficiency gains. Given this quick overview the code below should make sense but there are two functions that each iterate through the values of the same column in order to generate two additional columns.

Visualizing your AWS costs

Sun, 27 Nov 2016 00:00:00 +0000

There are a variety of cloud management services that connect to your cloud computing account and analyze your usage in order to offer recommendations that help improve efficiency, security, and reduce your costs. In fact, AWS even provides their own service, Trusted Advisor, that competes with the external vendors. Unfortunately, these vendors can get expensive quickly. The first useful tier of Trusted Advisor, categorized as Business, has a tiered pricing model based on your existing usage that starts at 10% of your AWS bill and decreases to 3% as you spend past $250k/month. External vendors are cheaper but can still get expensive depending on your bill: Cloudability starts at 1% of your AWS costs which compared to Trusted Advisor is significantly cheaper is still 1% of your AWS bill.

Recursive redirects with AWS Lambda

Sun, 13 Nov 2016 00:00:00 +0000

Two years ago I toyed around with an odd idea of implementing recursion over HTTP redirects. The idea is that the state is managed through the query string arguments and at each recursive step we just redirect to the URL for the next one. I still can’t think of a legitimate use case for this approach but have been on an AWS Lambda binge lately and wanted to see whether I can get this “redirect recursion” working under Lambda. Turns out it’s incredibly easy.

A poor man's data pipeline

Sat, 12 Nov 2016 00:00:00 +0000

Building a data pipeline can be a massive undertaking that typically requires deploying and configuring a Kafka cluster and then building appropriate producers and consumers that themselves come with dozens of configuration options that need to be tweaked to get the best possible performance. Beyond that one has to set up a coordination service, typically ZooKeeper, to handle a litany of concurrency and failure issues. These days having a data pipeline is a requirement for any data driven business but building a true streaming data pipeline entails a ton of dedicated effort.

Simple data visualizations from the command line

Wed, 26 Oct 2016 00:00:00 +0000

Lately I’ve been doing a variety of quick data investigations and they typically follow the same formula: write a query to fetch some simple data, copy and paste into Excel, do a minimal amount of manipulation, plot the results. Often this happens in a sequence where the results of one analysis leads to another one and so forth and so forth until the data has been sliced so many different ways that I’m able to figure out what I was investigating.

Revisiting my Twitter activity

Wed, 19 Oct 2016 00:00:00 +0000

While going through my old GitHub repos I discovered that the most starred repo was twitter-archive-analysis, a Python script that would generate a view visualizations of a Twitter archive. I haven’t touched the code in over 3 years and decided to see how it was holding up and whether any of it still worked. After a few false starts getting the necessary packages playing nicely together and updating the code to support Twitter’s new archive format, I was able to get the old code working. Compared to three years ago, the results are surprisingly not that different - I definitely tweet less frequently than I used to and my activity has shifted into being more about replies rather than general tweets.

Shaming meeting room hogs

Sat, 01 Oct 2016 00:00:00 +0000

One of the first things felt by a fast growing company is the lack of meeting space. The first few weeks at a new office it’s wonderful to know you can find a room whenever you need it. Yet after a few months and a bunch of extra people you realize you have to book meetings days in advance. And what makes this worse is seeing more than one room booked for the same meeting.

Supporting Disqus in AMP

Tue, 13 Sep 2016 00:00:00 +0000

After migrating my blog to AMP the last task was getting Disqus working again. The crux of the issue is that in order to improve page performance AMP disallows blanket script tags (which the Disqus integration leverages) but to make up for it comes with a variety of helpers to include officially support functionality. Examples of this include an amp-youtube tag to include YouTube videos and the amp-vimeo tag to include Vimeo videos. As a generic solution, AMP provides the amp-iframe tag which allows you to include a restricted iframe.

AMP migration scripts

Thu, 08 Sep 2016 00:00:00 +0000

Over Labor Day weekend I migrated my blog to use AMP but the first version was definitely a work in progress. One big item I needed to take care of was converting all my images to be AMP compatible by replacing <img> tag with <amp-img> along with the image width and height. I ended up writing a quick Python script to go through each of my posts, find each <img> tag, get the image’s dimensions, and then replace the original tag wit the AMP version. Unfortunately, I ran the script without too much testing and forgot to add closing tags which caused some of the content to go missing.

Food identification with Google's Cloud Vision

Mon, 29 Aug 2016 00:00:00 +0000

Something that I haven’t quite figured out is how to avoid wasting food. I like to think I keep good track of everything in my fridge but too often I end up finding something in the corner that spoiled and needs to be thrown out. Earlier today I was talking to someone at the office about this problem and how nice it would be if you could just have something that knows everything that’s in the fridge and can track how long it’s been there and an estimate of how long it will last. I’m sure refrigerators in 10 years will have this built in but I wanted to see what I could cobble together in an evening.

Writing scrapers as APIs

Wed, 24 Aug 2016 00:00:00 +0000

While building the Turo scraper I became annoyed that there was no API to make my job significantly easier. Then I wouldn’t have had to go through a variety of hoops and iterations to get the data I needed and would also not have to worry about changes to their page design breaking the script. This got me thinking about an idea to write my scraper in such a way that it’s exposed as an API. In that case I can architect the code so that the retrieval and manipulation of the ride data is completely separate from the scraping code. Then if and when Turo does decide to release an official API all I’d need to do is swap my unofficial implementation out for the official one.

Downloading your Turo ride history

Sun, 21 Aug 2016 00:00:00 +0000

I’ve been using Turo to rent our car out for the past couple of months and have been using a simple spreadsheet to track the revenue. Being a lazy engineer doing this manually became a bit tiresome so I finally automated it. Unfortunately Turo does not have a simple way of downloading the data and there’s no open API so I had to resort my usual solution: scraping. Luckily for me I just came off of updating my Yahoo fantasy football scraping script and was ready to do the same for Turo.

Integrating poorly documented Open Source libraries

Sun, 14 Aug 2016 00:00:00 +0000

Open source is great: if you find the right library you’re able to save a ton of time and get code that’s been through the gauntlet that you can confidently incorporate into your system. Unfortunately many open source libraries are partially baked with documentation that doesn’t always accompany the rapid development of the code. This leads developers to repeatedly cross reference their code with some archaic documentation and then wonder why it’s not working as expected. This is proportional to the obscurity of the library - popular libraries will have most of their kinks worked out but esoteric ones that are likely maintained by one person won’t have the polish.

Fantasy football stats: 2016-2017 edition

Sat, 13 Aug 2016 00:00:00 +0000

This is an annual tradition now but I just updated my old script that crawls and extracts the projected fantasy football data from Yahoo to work with the 2016-2017 season. The changes were incredibly minor: Yahoo broke the the login page into two steps and there was a minor change in the order of the columns. Both of these were trivial to implement and the code is up on GitHub. If all you care about is the raw data you can just download the CSV.

One of my favorite programs

Sat, 06 Aug 2016 00:00:00 +0000

While working on a small programming puzzle I remembered Peter Norvig’s spell checker and how blown away I was after seeing it for the first. It’s one of my favorite examples of code that’s clean and elegant while being extremely expressive and powerful. If you haven’t seen it yet I encourage you take a look and step through it since he does a much better job of explaining both the code and theory than I ever could.

Coding puzzle: Word transformation through valid words

Sun, 17 Jul 2016 00:00:00 +0000

A fun engineering puzzle I heard this week was to write an algorithm that finds the shortest path between two words of the same length where you’re only allowed to change a single letter each step and every word needs to be valid. This morning I decided to have some fun with it and wanted to jot down my thought process going through the exercise in the hope that it provides a bit of perspective on how I approach code.

Maximize the potential energy of your code

Mon, 04 Jul 2016 00:00:00 +0000

Potential energy: the energy of a body or a system with respect to the position of the body or the arrangement of the particles of the system.
Dictionary.com

Kinetic energy: the energy of a body or a system with respect to the motion of the body or of the particles in the system.
Dictionary.com

Analyzing my blog

Sun, 12 Jun 2016 00:00:00 +0000

I started actively blogging in 2013 and have been consistently writing 2 posts a week. There’s a ton of information here and I spent some time learning R all over again in order to analyze and visualize my blogging history. I started with a simple Python script that went through each post and dumped it into a CSV file with a series of columns that would be easy to analyze via R. The columns ranged from numeric stats - such as how many words, tags, images, and links - to the actual text of the post itself. The goal was to put in a structured enough shape that the rest of the analysis could be handled in R. I started by collecting some summary statistics and looking at them over time but got carried away and ended up digging deeper into my evolution as a blogger.

Word clouds in R

Mon, 06 Jun 2016 00:00:00 +0000

Analyzing my blog is taking longer than expected but my goal is to have something meaningful over the weekend. In the meantime I wanted to share a quick script I discovered to generate a word cloud in R. I remember doing this years back in D3 and having to spend a bunch of time figuring it out. Compared to that doing it in R is a breeze. In this case I have a CSV dump of my blog in /tmp/out.csv and am generating two word clouds - one for keywords and the other for tags of my blog posts.

Analyzing IMDB data: Actors vs actresses

Sun, 22 May 2016 00:00:00 +0000

After getting the IMDB data loaded it was time to dive in and start looking at the data. In 2012, I did an analysis to examine the way actor and actress ages have changed over time. Unfortunately I did not have a large dataset and had to write a quick crawler to look at the top 50 movies during each decade. This time around, and with the help of CuriousGnu, I was able to get my hands on a much larger dataset. After cleaning and filtering the data I was left with over 208,000 unique actors (~65%) and actresses (~35%) spanning over 371,000 movies. The code is up on GitHub and contains both the queries used to pull the data from MonetDB, the R code to generate the charts, and a small script that generated the animation below. If you have suggestions or ideas definitely let me know.

Analyzing IMDB data: Step 1 - Cleaning and QA

Sat, 21 May 2016 00:00:00 +0000

In 2012 I did a simple analysis of IMDB to analyze the change in actor and actresses’s ages over time. At that point I limited the analysis to the top 50 movies each decade and hacked together a quick script to crawl and scrape the IMDB analysis. A couple of weeks ago I came across a great post by CuriousGnu that did a similar analysis across a larger set of movies but limited to movies since 2000. I reached out and they were kind enough to give me a DigitalOcean instance containing the data already loaded into MySQL. The analysis should be finished up tomorrow but I wanted to write this post up to share the mundane parts of the process. The janitorial part is critically important to an analysis and it’s important to get it right or the results will may be meaningless or even completely wrong. The NY Times interviewed a variety of data scientists and came away with the conclusion that 50 to 80 percent of a data scientist’s time is spent cleaning the data. This is no exception and I wanted to provide a sense of the effort and thought that goes into getting data into a state that’s actually useful.

Identifying unused database tables

Wed, 11 May 2016 00:00:00 +0000

When writing code it’s very easy to accumulate deprecated database tables that end up as zombies - they’re still around and may even be populated and used by a variety of side scripts but if they disappeared and the dependent code was removed nothing would be different. In fact you’d have a smaller code base, a smaller database, and would hopefully improve everyone’s productivity a tiny bit.

Dealing with the tables are are still being populated and read requires a bit of investigative work and knowledge of the product since there’s no simple way of identifying them. But there are a simple ways to identify tables that are no longer updated.

A Telegram blog bot

Sat, 23 Apr 2016 00:00:00 +0000

A combination of bots being in vogue and Telegram offering $1M in bot prizes got me to spend a little bit of time writing a bot last week. To get my feet wet I created a simple, self-serving bot that would reply with a random blog post when sent a /blogme command. The code itself is extremely straightforward and most of the time was spent going through the Telegram bot docs and getting the deployment and HTTPS setup. A nice feature that Telegram has is the ability to write a bot that can respond to both polling and webhooks. The polling approach is a much trivial to get started with since you don’t need to worry about any of the devops work and can work on the core interaction. The cons are that it won’t respond immediately and you need a way to track messages your bot has already replied to. Changing it to a webhook provided real time responses but made it a bit more difficult to test and wrapping everything inside a minimal web framework. The biggest hiccup was the requirement of HTTPS for a webhook integration but Let’s Encrypt made it simple to get up and running. A year ago I wouldn’t have bothered prototyping anything that required HTTPS but these days it’s incredibly easy to set up. The code is up on GitHub and if you’re interested in bots definitely take a look. And if you have Telegram installed try messaging “danblog” with /blogme to get a random blog post.

Unintended consequences

Sun, 10 Apr 2016 00:00:00 +0000

Earlier today I read an article about MaxMind, a company that offers an IP address to geographic location mapping service, making a seemingly minor decision in 2002 that that led to unintended consequences that have been going on since then. The article goes into detail about the decision and the effect but the main idea is that it’s not a prefect system and they needed a way to approximate some IP addresses to particular locations. Lo and behold these locations are now seeing tons of harassment from law enforcement and various strangers online.

Generalize at n=3

Thu, 07 Apr 2016 00:00:00 +0000

Engineers strive to write code that’s general and flexible enough to adapt to support a variety of cases with minimal changes. Unfortunately, writing general code isn’t easy and requires significant thought, effort, and experimentation. The challenge is figuring out the appropriate time to generalize your code.

If you do it too early you may spend unnecessary time writing generalized code that will never be used again. Even worse you may write code that you think is generalizable but ends up collapsing under its own weight under future scenarios. In this case writing minimal code would have served you better since it would have been much easier to adapt or throw away to support the new case.

The MySQL enum type

Thu, 10 Mar 2016 00:00:00 +0000

The MySQL enum field provides a nice compromise - the space efficiency of using an integer, the human readability of text, and basic type safety. Yet I had this vague recollection of reading something that made it seem enums carried a ton of risks when changing the column definition so wanted to see if I could “break” it. Turns out it’s a lot more resilient than I thought. I went through a series of combinations - ranging from changing the order of the enums in the definition to trying to insert values that didn’t exist but in every case it handled it as expected. Doing a bit of research I discovered how MySQL represents the enum type. Rather than storing the values in a specific order MySQL supposedly creates a map-like structure to relate the integer values with their enum counterparts. This allows you to change the order of the enum definition without changing the underlying map or any of the stored values. I still wouldn’t use enums for anything that would require a join but for storing small and simple sets of data it works great.

Cleanest way to read a CSV file with Python

Sun, 10 Jan 2016 00:00:00 +0000

Python’s my goto language for doing quick tasks and analyses with the majority of them being quick scripts to analyze a file or pull some data. I’m constantly looking to improve my code and lately have developed the following approach. The goal isn’t to make it as short as possible but to make it as expressive and clean as possible. They’re related but not synonymous.

#!/usr/bin/python

import csv
from collections import namedtuple

# Can add whatever columns you want to parse here
# Can also generate this via the header (skipped in this example)
Row = namedtuple('Row', ('ymd', 'state', 'size', 'count'))

with open('file.csv', 'r') as f:
 r = csv.reader(f, delimiter=',')
 r.next() # Skip header
 rows = [Row(*l) for l in r]
 # Do whatever you want with rows

Jersey City garbage truck routes
Sat, 12 Dec 2015 00:00:00 +0000

 
 
 
 
 

A couple of months ago I took a stab at plotting the Jersey City parking zones after getting frustrated that the only place to see them was a PDF of streets and addresses. Last week someone left an awesome comment pointing out that Jersey City has a bunch of open data available, including a near-real time feed of garbage truck locations, a general open data portal, as well as the ability to request custom data. As a first project I decided to capture the movement of the garbage trucks every minute and then plot the results on a map. The results are interesting - some trucks remain local to Jersey City while others end up venturing as far as Newark Airport. The final visualized routes are at https://dangoldin.github.io/jersey-city-open-data/ and the code is up on GitHub.


Think interfaces, not implementation
Wed, 02 Dec 2015 00:00:00 +0000
An idea I’ve been preaching over the past few days is to start thinking in terms of interfaces when thinking about writing code rather than the actual implementation. It’s a higher level of abstraction that leads to a higher quality and more scalable product. Rather than focusing on the details it’s better to think about the components and how they’ll interact with another - this also makes it easy to put in a crappy implementation for now while making it easy to modify and rewrite in the future. As engineers there’s a strong desire to obsess over the perfect code which can lead to a significant amount of refactors and rewrites without translating into actual business value. Thinking in terms of interfaces and components forces you to get the design and architecture right and leaving the implementation details for later. A side benefit for me has been being able to take pride in the design and flow and not worry about the code itself - allowing me to write code at a much faster place and sprinkle a series of todos for the parts of the code that I know need improving.


More MySQL fun
Sat, 21 Nov 2015 00:00:00 +0000
I had a bit of fun with MySQL earlier this week when trying to explain a non obvious “group by” behavior. It’s fairly common to want to manipulate a field in order to transform it into something more useful. The difficulty arises when you want to keep the original name. Below is some SQL code that highlights the odd behavior.
drop table if exists dan_test;

create table dan_test (
 id int not null,
 id2 int not null
);

insert into dan_test (id, id2) values (1,1), (2,2), (3,3);

select * from dan_test;

select id, case when id = 1 then 2 else id end as id, id2
from dan_test;

select id, sum(id2)
from dan_test
group by id;

select case when id = 1 then 2 else id end as id, sum(id2)
from dan_test
group by id;

select case when id = 1 then 2 else id end as new_id, sum(id2)
from dan_test
group by new_id;

My old projects
Thu, 12 Nov 2015 00:00:00 +0000
Writing up my old projects got me browsing through my GitHub account to see what else I’ve worked on. Some I’ll update when I get a good idea while others I completely forgot until going through the list. I noticed two big themes when going through the list. The first is how much nicer it is to have projects that are in static HTML/CSS/JavaScript since they can be hosted publicly on GitHub and don’t require any setup or configuration to start using. The other is how many third party libraries or APIs I’ve used and how much more difficult everything would have been had I had to build everything from scratch. If anyone is interested in forking and ressurecting some of these I’ll be glad to polish it up.


Jsonify.me 2.0
Wed, 21 Oct 2015 00:00:00 +0000
A couple of weeks ago I wrote about the idea of having a “go to” project that you use to pick up a new language and earlier this week I finished the bulk of the rewrite of jsonify.me. It went through a Node.js phase, a Scala phase, and is currently in the go phase. The idea is to give people an open ended and simple way to generate a personal JSON object, similar to how people may have an about.me page but in JSON. This object can then be mapped to any subdomain (mine is at json.dangoldin.com) and be referenced by any third party code. For example, you can construct your personal jsonify.me object based on the information in your various social media profiles and then make that information accessible to a variety of sites or pages that can generate it in a variety of ways. One site can turn it into a simple resume while another one can turn into a visual timeline of your history. At the moment it’s entirely open ended with the vast majority of the functionality provided solely through an API. Over time I’ll add some more bells and whistles but I’d love to see the community come up with their own unique JSON format that can then get adopted - similar to the way the hashtag system on Twitter evolved. I suspect it’s going to be significantly more complicated since there’s no 140 character limit but am still interested to see where this goes. Play around with it and let me know what you think!


Dates in the shell
Mon, 19 Oct 2015 00:00:00 +0000
The longer I code the more I appreciate the power of the shell. Getting familiar with common commands is a great way to improve your productivity and over time you amass a massive collection of scripts that allow you to do nearly everything. The most recent utility I discovered was “date”. As expected, it displays the current date and time but it can easily be adapted to display the current datetime in nearly any date format but also allows you to offset the current date in a variety of ways.


Mapping the Jersey City parking zones II
Thu, 24 Sep 2015 00:00:00 +0000

 
 
 
 
 

I finally had the chance to finish up the Jersey City parking zone mapping project from a couple of weeks ago. The goal was to take a PDF of valid addresses for each zone and visualize it on a map. The result can be found at https://dangoldin.github.io/jersey-city-open-data/ and includes the zones that had enough geocodeable addresses to generate a valid polygon.


Mapping the Jersey City parking zones
Sat, 12 Sep 2015 00:00:00 +0000
A big part of owning a car in Jersey City is dealing with the street parking. Unfortunately, Jersey City does not make it easy to see what the zones are - instead there’s a PDF that lists the streets and address ranges that are part of each zone. After getting frustrated with this annoyance for too long I decided to just take matters into my own hands and visualize the zones through some scripting. This is a relatively simple project that still involved some false steps so I wanted to document the process and provide a peek into my development approach.


Reprioritizing a non priority RabbitMQ queue
Wed, 12 Aug 2015 00:00:00 +0000
Earlier today we had a hiccup where we had a bunch of messages piled up on a RabbitMQ queue that were not being consumed. Some of these tasks were very quick data loads while others were more involved jobs that could take multiple minutes to run. Normally these are distributed relatively evenly across the day so it’s not a problem but in this case we had hundreds of tasks in a random order and we wanted to shuffle them around such that the data load tasks executed first so that the data would be quickly accessible to other higher priority jobs.


The Go interface
Wed, 29 Jul 2015 00:00:00 +0000
I’ve only been playing around with Go for a couple of weeks but one of the language design decisions I’ve really enjoyed is how interfaces are handled. Coming from a traditional object oriented background it’s typical to define an interface that defines a few method signatures and then explicitly implement that interface in a new class. Below’s a trivial example of this approach in Java:
interface Animal {
 public boolean isFurry();
 public String speak();
}

class Dog implements Animal {
 public boolean isFurry() {
 return true;
 }

 public String speak() {
 return "Woof";
 }
}

public void aRandomFunction(Animal a) { .. } // Can take anything that implements Animal

Comparing SQL schemas
Sun, 12 Jul 2015 00:00:00 +0000
During development it’s common to get your dev database out of sync with the one in production. Sometimes it’s due to an additional column in development you added before realizing it wasn’t necessary and other times it’s just creating a few temporary tables on production that you forget to drop. In both cases it’s useful to reconcile the schema differences every once in a while to keep your database in a clean state. In the past I would just run a simple query (select table_schema, table_name, column_name from information_schema.columns;) on each environment and then use either Excel or Google Sheets to spot the differences. This takes a bit of time so this weekend I put together a quick JavaScript tool to automate the process. You simply run the schema query on each of the environments and paste the resulting rows into the two text areas. The result is a JSON based diff showing the additions, deletions, and modifications to each of the tables and fields. The next step is to modify it to also identify differences in the column types.


Domain specific API definitions
Tue, 23 Jun 2015 00:00:00 +0000
Yesterday, Amazon announced a major update to their Python client, boto3. The core functionality is unchanged but they used a clever solution to make it easier to add, modify, and remove endpoints. By coming up with a standardized representation for each of the endpoints they’re able to write wrappers in different languages that generate the API calls programmatically. For example, I’ve included a subset of the EC2 definition below. It contains the information necessary to programatically generate the API wrapper to hit the appropriate EC2 endpoints.


Date range generation
Sat, 30 May 2015 00:00:00 +0000
I finally had the chance to go back and add another quick tool to my JavaScript arsenal. This one lets you specify a start date, an end date, a step size and interval, along with a desired date format and it will generate the dates in between. This is a surprisingly common activity for me. Every time I need to split a query into multiple date ranges or come up with a series of arguments for various jobs I end up using Excel to come up with the appropriate date ranges. By having it available via the web it makes it a lot easier to generate exactly what I need as well as provides the flexibility to keep on improving. If there are any improvements you’d like to see or if anything is unclear definitely let me know.


Adding columns in PostgreSQL and Redshift
Thu, 23 Apr 2015 00:00:00 +0000
A frequent event when working with a SQL database is adding a column. Ideally, you’d want to add this column before or after another one that makes sense rather than all the way at the end. MySQL makes this straightforward since you can use the AFTER keyword when adding a column to specify exactly where it should be added. PostgreSQL and Redshift make this difficult since all new columns are automatically added at the end.


Redshift meets Excel
Mon, 06 Apr 2015 00:00:00 +0000
As part of our data pipeline, we have a Redshift agg job that takes low level data and rolls it up to an hourly aggregate. A latter job takes the hourly data and rolls it up to a daily level which is used for high level reporting and summary statistics. Earlier this week we ran into a hiccup that caused some of these aggregate jobs to fail. After fixing the issue we had to figure out what data was affected and rerun it. We wrote a simple query to count the numbers of rows per day per hour in order to spot any gaps.


Getting the most out of log4j
Sat, 28 Feb 2015 00:00:00 +0000
Something that’s incredibly helpful when writing Java code is customizing log4j. There are a variety of configuration options and learning just a little bit about them can make you notably more productive. I’ve found two features that have sped up my development cycles.
One was updating my PatternLayout to include the filename and line of each message. With Eclipse, this allows me to quickly jump to the relevant code block whenever anything looks odd rather than having to first open the file and then search for that particular message.


Lists and localStorage
Thu, 26 Feb 2015 00:00:00 +0000
I recently discovered the localStorage functionality in HTML5 and used it on a quick internal tool at TripleLift. One hiccup I ran into was that while it provides the ability to set and get key/value pairs it stores everything as a string so I needed to write a few utility methods to get it to work with lists. They’re pretty straightforward but hopefully they inspire someone to improve on them.


URL redirection app
Sat, 07 Feb 2015 00:00:00 +0000
At TripleLift, we’re big fans of the Switcheroo plugin and rely on it during development to test new versions of our code. It allows us to override a production hostname with one of our development boxes so we can see how our code works on a live site. So if a production site is referencing a JavaScript file at http://production-environment/script.js we use Switcheroo to have it reference the development file at http://dev-environment/script.js. Unfortunately, it’s only available for Chrome which makes it more difficult to run browser specifics tests on other browsers.


Fun with GitHub's map tools
Sun, 18 Jan 2015 00:00:00 +0000
After discovering GitHub’s map visualization feature I needed to give it a shot on the only GPS dataset I had available, my runs from RunKeeper. Unfortunately, the RunKeeper files were in GPX while GitHub expects either geoson or topjson. A short Python script later and I was able to convert the GPX data into geojson. The other hiccup I encountered was that the generated geojson file was too large for GitHub to visualize. My 232 runs contained 162,071 latitude/longitude pairs which turned into a 4MB file - not massive but large enough for GitHub to refuse to visualize it. The simplest solution was to generate multiple files but that made it impossible to see all my runs on a single map. The other solution was to see if converting to topojson would reduce the file size. That helped but I wasn’t able to find the right balance between compression and quality and ended up with a hybrid approach - two files, one per running year, each in topojson.


2014 stats
Sun, 11 Jan 2015 00:00:00 +0000
At the beginning of last year I decided to do my part of the quantified self movement and started a daily log of how much I’ve slept, my mood during the morning, afternoon, and evening, as well as what I ate and drank. There were a couple of stretches where I forgot to fill in the details and did what I could from memory. My mood also had a pretty big impact on the way I filled in the subjective questions but hopefully it balances out over a year. I tracked the data via a Google spreadsheet and exported it as a CSV in order to analyze it via a simple Python script. For now I’ve only pulled some summary stats but will take a deeper look in the next couple of days to examine the distributions and identify any patterns.


Redirect recursion
Wed, 31 Dec 2014 00:00:00 +0000
I’ve stumbled onto what seems to be a solution without a problem but something that’s been fun to experiment with and might have an actual application. The idea is to replace a recursion step with a URL redirection. In this situation the base case will return a 200 response while the recursive step will do a redirection with a slightly updated URL. The sample node server below uses this idea to handle a three tasks - sum up to n, compute a factorial, and test whether an integer is prime.


Some simple AWS tools
Sun, 09 Nov 2014 00:00:00 +0000
Last night I took an old bash script I wrote that simplified connecting to an EC2 instance in an AWS account and implemented the same code in Python. The old code worked by listing a set of AWS instances and then prompting to pick a single one to connect to. The problem was that it wasn’t always easy to find the index of the desired instance and the code took a bit of time to run.


Top down vs bottom up coding
Tue, 16 Sep 2014 00:00:00 +0000
Over the years, I’ve noticed two distinct coding styles. Some approach problems top down and will stub out the entire solution using dummy values and methods and come up with a naive solution before fleshing everything out properly. Others will instead take a bottom up approach and try to complete each method entirely before moving on to the next one.
Especially for larger problems, I prefer the top down approach. By stubbing out the various pieces it’s easy to see how everything fits together and makes it easy to identify and solve potential issues before investing a ton of effort into a poor implementation. The other benefit is that I start thinking at a systems level and come up with implementations that tend to be more extensible.


Managing settings files in Django projects
Sat, 30 Aug 2014 00:00:00 +0000
I was helping a friend deploy a Django project over the weekend and we chatted about the best way to manage multiple settings files in a Django project. The primary reason is that you will typically have different settings between a production and development environment and but at the same time will have a lot of options shared between them. A production environment will typically be more restrictive and optimized for performance whereas a development environment will be setup to provide as much debug information as possible.


Yahoo fantasy football stats - 2014 edition
Tue, 26 Aug 2014 00:00:00 +0000
This might be too late for some but I dug up my Yahoo fantasy football stats scraper from last year and updated it to work for the 2014 season. The old version used the great Scrapy framework but unfortunately Yahoo changed something on their end that made the login spoofing too difficult to do via a backend script. The new approach uses Selenium to open up a Chrome web browser, login to Yahoo, and then iterate through each page of stats and downloads the data into a CSV file.


Set up HTTPS on EC2 running Nginx without ELB
Tue, 15 Jul 2014 00:00:00 +0000
I recently needed to set up HTTPS for my side project, better404.com. Amazon makes it easy to set up by uploading it directly to an ELB but in my case it’s hosted on a single AWS instance so I didn’t want to pay for an ELB that would be more expensive than my one instance. I’ve heard horror stories and expected the worst but it turned out surprisingly easy. Hopefully these steps can help someone else out.


Retrieving a Twitter user's followers and followees
Mon, 07 Jul 2014 00:00:00 +0000
After reading Gilad Lotan’s post where Gilad bought 4,000 Twitter followers in order to analyze them, a friend of mine was inspired to analyze his followers to see if he could get any insight and come up with a neat visualization. The first step was downloading a dataset containing his followers and followees as well as the followers and followees for each of those accounts - the idea being that by going two levels deep you see how similar the various accounts are to each other based on who and what they follow and whether there are any patterns.


Debugging a reverting database update
Sat, 07 Jun 2014 00:00:00 +0000
I ran into an odd bug today where a database entry was reverting itself after a seemingly simple update. For Better404, a customer can change the design of their 404 page but it turns out that every once in a while a change would go through but within a minute would revert back to the previous value. At the same time, update queries run directly via the MySQL client ran fine and were not being reverted - just the ones made through the site. To see what was going on, I enabled full query logging in MySQL (SET GLOBAL general_log = ‘ON’) and sure enough I saw a lagging query that would update the record to the prior values. Stepping through the code I was able to figure out the cause.


Generate fake SQL data using JavaScript
Thu, 29 May 2014 00:00:00 +0000
A problem I occasionally run into is needing to generate a bunch of fake data and insert it into a database table. My usual approach has been to generate this data in Excel and then use a series of string concatenations to generate the necessary insert statements which I’d then execute in the SQL client. After doing this one too many times I decided it was time for a better, more automated approach and hacked one together in JavaScript. It’s currently a part of my js-tools GitHub repo and suggestions are welcome. One thing I definitely need to add is the ability to specify the range of possible values for each field rather than using a hardcoded distribution.


Examining ssh login requests
Fri, 16 May 2014 00:00:00 +0000
I recently migrated to Digital Ocean and spent some time beefing up its security. One of the things I looked into was the various SSH attempts being made and to see if there was a pattern. Luckily, I’m running Ubuntu and every SSH attempt is logged by default to /var/log/auth.log and all it required was a quick one liner to see the failed attempts by username.
grep "Invalid user " /var/log/auth.log | cut -d' ' -f8 | awk '{a[$0]++}END{for(i in a)print i,a[i]}' | sort -k 2 -n -r | head -n 100

Most commonly used shell commands
Mon, 12 May 2014 00:00:00 +0000
I spend a large chunk of time working in the terminal and was curious to see what my most commonly used shell commands were. This also gave me an opportunity to practice writing one liners and learn a bit of awk.
history | cut -d' ' -f4 | awk '{a[$0]++}END{for(i in a)print i,a[i]}' | sort -k 2 -n -r
The script is simple - look through my command history, extract the first word, and count the number of times that word appears. I was surprised to see git at the top but it makes sense - I tend to run it as a sequence (git status, git commit, git push) so it leads to an inflated count. The rest make sense - they’re a mix of the standard navigation commands as well as command related to my current projects. Next step is to set up a cron job to track this usage over time and see how it changes.


Gap fills and cross joins in Excel
Sat, 03 May 2014 00:00:00 +0000
During my consulting years I’ve done a ton of Excel and noticed people getting frustrated by two seemingly simple operations. The first is getting a worksheet with gaps in a column and needing to fill it with values from the cells above and the second is doing a cross join between two sets of values.
The solution to the gap filling can be done by explaining the solution in such a way that it can be implemented via an Excel formula. The best I could come up with is “If a gap is a value, take the value of the closest non empty cell above it, otherwise keep its value.” We can create a formula in another column that takes this approach and after coming up with the new cell values and pasting them over the originals. In the image below, the formula in cell D2 is =A1 and the formula in D3 is =IF(A3="",D2,A3) with D4 down being relative copies of D3.


Vertical integration and web development
Wed, 16 Apr 2014 00:00:00 +0000
Lately, I’ve been thinking about tightly coupled systems and how prevalent JavaScript has become on the web.
Tightly coupled systems scare me. They will undoubtedly break and bring down big chunks of your infrastructure. The solution is to think about your system in terms of various independent services that are responsible for only doing a few things well that won’t bring down the rest of the system if they fail. This approach makes it easier to maintain your code as it grows and also reduces the risk of massive failure. The challenge is figuring out how to break your project down into these services and being sure to revisit that decision as you grow.


PostgreSQL Fibonacci
Fri, 04 Apr 2014 00:00:00 +0000
Earlier today I was researching whether it was possible to generate Fibonacci numbers using a SQL query. A Google search turned up a short PostgreSQL query that uses a recursive approach. Since this is recursion, the query starts by defining a base case and then goes on to define a generation step with a stopping limit.
with recursive f as (
 select 0 as a, 1 as b
 union all
 select b as a, a+b from f where a < 100000
) select a from f

More Sierpinski fun
Fri, 21 Feb 2014 00:00:00 +0000
As a follow up to my previous post, I modified my Sierpinski generation code to allow specifying the number of sides and the distance ratio for each iteration of the loop. The Sierpinski triangle can be generated with 3 sides and a distance ratio of 0.5. Increasing the number of sides and decreasing the ratio leads to some interesting patterns - it looks as if for a given N, we get N shapes each consisting of N shapes. I suspect this is a fractal pattern - similar to the triangle - but it’s difficult to confirm given a fixed screen resolution. I’d love to know what’s going on here and whether there’s a relationship between the number of sides and the distance ratio.


Sierpinski triangle in D3
Wed, 19 Feb 2014 00:00:00 +0000
There’s a little known algorithm for constructing a Sierpinski triangle that is surprisingly easy to implement.

Start the three vertices that form a triangle
Pick a random point inside the triangle
Pick a random vertex
Go halfway from a the random point to the vertex and mark that point
Go to step 3 using the result of 4 as the starting point

I’m trying to get better at D3 and thought it would be a good exercise to code it up. The resulting image is below (generated using 10,000 points) and the JavaScript is in the following file. Next up is to write a new script that allows a user to specify the number of vertices and the adjustment factor - the Sierpinski carpet can be generated with 4 vertices and a distance adjustment factor of a third rather than a half.


Using virtualenv in production
Mon, 10 Feb 2014 00:00:00 +0000
One of my favorite things about Python is being able to use virtualenv to create isolated environments. It’s extremely simple to use and allows you to have different versions of Python libraries used by different projects.
The thing that’s tricky is getting virtualenv set up on a production environment under different services since each one requires a slightly different configuration. I’ve gone through my projects and collected the various ways I’ve gotten it running for different services. I’m sure I could have done it differently but the following worked for me and will hopefully come in handy to others. If you have any questions or I’m not being clear enough let me know and I’ll updat the post with more information.


Visualizing GPS data in R
Wed, 05 Feb 2014 00:00:00 +0000
Earlier today I read Nathan Yau’s post that had a quick R script to plot GPX file data onto a map. I was able to quickly load up my RunKeeper data from 2013 and came up with a pretty cool visualization of each of my outdoor runs. Since my runs occurred across multiple cities and continents the visualization turned out to be very sparse without a great sense of where the runs were. I made a two quick changes to the script to make it more useful for my data: a map overlay to see where in the world I ran and an ability to view a zoomed in area of the map. I’ve included the updated script and the resulting plots below.


Solving coding tests in PostgreSQL
Sat, 25 Jan 2014 00:00:00 +0000
Most developers are familiar with the FizzBuzz code test which is a quick way to filter out developers who can’t code. At Yodle, we had our own, slightly more challenging problem. The challenge was read in a text file and then print out the frequency each word appears in descending order. It’s more complicated than FizzBuzz but it assesses a variety of skills. The solution needs to do the following:


Developing on a remote instance
Thu, 23 Jan 2014 00:00:00 +0000
One of the first things I was given when joining TripleLift was a Macbook Air and an Amazon EC2 instance to do my development work on. Before that, every company I worked at would give me a pretty powerful computer so that I’d be able to do my development work locally. At first, coding on a remote instance took some getting used to but now I’m a fan of this approach.


Taxi prices around the world
Thu, 09 Jan 2014 00:00:00 +0000
I initially set out to add some visualizations to an earlier post comparing taxi fares between NYC and Mumbai based on some reader suggestions. After a few visualizations, I wasn’t discovering anything new and decided add taxi fare data from other cities to make it more interesting. I ended up simulating rides in different cities on worldtaximeter.com and combining that with the data from taxiautofare.com and www.numbeo.com in order to break down each city’s fare into a base fare, the included distance, the rate per local distance unit, and the rate per minute. Since each city’s fare came in local units I also had to convert to miles (sorry world) and US dollars (sorry again). Using R we generate the fares for the various combinations of distances and stoppage times and start diving into the data. As usual, the data and code are up on GitHub with contributions, corrections, and suggestions welcome. I’d also love to get the real rates for the cities so either do a pull request or let me know what they are in the comments and I’ll update the post.


Visualizing RunKeeper data in R
Sat, 04 Jan 2014 00:00:00 +0000
What better way to celebrate running 1000 miles in 2013 than dumping the data into R and generating some visualizations? It’s also a step in my quest to replace Excel with R. I’ve included the code below with some comments as well as added it to my GitHub. If you have any ideas on what else I should do with it definitely let me know and I’ll give it a go.


Using the information_schema.columns table
Sun, 15 Dec 2013 00:00:00 +0000
Something that’s been really helpful to me in understanding a MySQL database is the built in information_schema.columns table. It provides information on every column in the database and is queryable just like any other table. This makes it easy to quickly find all tables that have a particular column name or all columns that are the same data type. There have been countless times where I knew the data existed somewhere but couldn’t recall which table it was in. Querying the information_schema.columns table for the foreign key helped me quickly figure it out. Below are some sample queries that retrieve data from the information_schema.columns table:


Drowning in JavaScript
Sun, 01 Dec 2013 00:00:00 +0000
I recently installed Ghostery and am amazed by the number of JavaScript libraries being loaded on the sites I visit. Almost every site I visit has at least one analytics library, a few advertising libraries, and some social network sharing libraries.
To be a bit more quantitative, I pulled the libraries used by 20 of top sites to see if anything stood out. The biggest surprise was how differently the various types of sites used these libraries. Every single publisher used DoubleClick and yet only a quarter of them used Google Analytics while 80% of the social networks I looked at used Google Analytics and only 40% used DoubleClick. The other interesting piece was how many more libraries an average publisher uses compared to a social network or ecommerce site. Five of the 13 publishers I looked at included at least 20 JavaScript libraries while the most libraries included by a social network was 4, which was Pinterest. The bulk of these additional libraries tend to be advertising specific so it’s not that surprising that publishers have more of them but the difference in volume was shocking. I’ve included the data at the bottom of this post in case someone wants to take a stab at it but something on my todo list is to automate the process of gathering this info rather than relying on Ghostery and copy and paste. Once I get get it done I’ll follow up with another post analyzing the larger set of data.


Genetic programming Connect 4
Sat, 30 Nov 2013 00:00:00 +0000
Over Thanksgiving break I was going through some old GitHub repos and found an interesting one I wanted to share. It’s a Connect 4 bot that’s evolved through a genetic program. The goal of the strategy is to choose a column to move to that will give the highest probability of a win given a board position. To figure out the move column, the genetic program simulates play of strategy against strategy and gives the most successful ones a greater chance of reproducing into the next generation. The idea is that over time the resulting strategy will be the most fit.


RDS and R
Fri, 15 Nov 2013 00:00:00 +0000
In my quest to replace Excel with R I’ve been spending the past week trying to do everything in R. It hasn’t been that easy with many things taking longer due to me having to reference the R docs but one thing that’s been great so far is being able to quickly run a query on Amazon’s RDS and pull data into a data frame for quick analysis. Being able to wrap this into a reusable function makes things even better. The one thing that makes it tricky was not being able to connect to RDS directly but having to tunnel through an EC2 instance. Below are the steps to replicate the setup.


Some JavaScript Tools
Sat, 05 Oct 2013 00:00:00 +0000
Over the course of this year, I’ve been writing two posts a week and been running into various formatting/design issues, two of which I finally dealt with earlier this week. One was embedding an Excel table into a blog post and the other was creating a BCG style “growth-share” matrix.
To convert a table from Excel to HTML I would write Excel formulae that would wrap each cell in a <td> tag and then wrap each row in a <tr>tag. I’d then copy and paste the result into the text editor to add the header row and finish up the styling. To generate a growth-share matrix, I’d just use Google Drawing or Keynote to draw the axes and labels before taking a screenshot and cropping it into a square.


Extract info from a web page using JavaScript
Mon, 26 Aug 2013 00:00:00 +0000
How many times have you tried copying something from a webpage into Excel and discovering that the formatting got completely messed up and forced you to clean the data up manually? With just a bit of knowledge about HTML and CSS you can use JavaScript to get the information you want without having to struggle with the formatting issues.
In my case, I participated in a fantasy football draft and wanted to share the list of players I drafted with a friend. Unfortunately, copying and pasting didn’t work so I decided to jump into JavaScript. Hope these steps give a sense of how to approach a simple scraping problem. The idea is to use the browser’s inspect element feature to find the pattern that the element we’re interested in have in common. Then, we use JavaScript to find the elements matching that pattern and extract the information we want.


Run Django under Nginx, Virtualenv and Supervisor
Tue, 30 Jul 2013 00:00:00 +0000
After yet another attempt to deploy a Django application I decided to document the steps required to get everything up and running. The tutorials I’ve seen tend to focus on individual pieces rather than on the way all these packages work together which always led to me a lot of dead ends and StackOverflow so this will hopefully address some of those issues.
In particular, I want to focus on the configuration rather than the installation of the various packages since that’s covered in the package documentation.


Scraping Yahoo fantasy football stats with Scrapy
Wed, 17 Jul 2013 00:00:00 +0000
Last week, someone reminded me of an old project I had on GitHub that scraped fantasy football stats from Yahoo. Unfortunately, it was antiquated and failed to retrieve the data for the current season. I’ve also been interested in trying out the Scrapy framework and decided this would be a good opportunity to give it a shot. I tried finding a sample project that dealt with authentication as a starting point but wasn’t able to find one so hopefully my attempt can serve as an example to others.


Fun with Prolog: Priceonomics Puzzle
Fri, 07 Jun 2013 00:00:00 +0000
The Priceonomics blog is one of my favorites so when I saw that they had a programming puzzle up I decided to have some fun with it. And what’s more fun than hacking around with a quirky, esoteric programming language? I remember having fond memories of playing around with Prolog in middle school so decided to dig it up again in an attempt to solve this puzzle.


Citibike Directions: Second Attempt
Tue, 04 Jun 2013 00:00:00 +0000
To coincide with the launch of Citibike, I wrote a simple web app that provided cycling directions from one Citibike station to another. The biggest piece of feedback I received was that people care about getting from place to place rather than from one Citibike station to another. Based on this feedback, I updated the app to provide directions from any New York City address to another by breaking every trip down into three steps: the first is to walk to the nearest Citibike station, the second is to bike from one station to another, and the last is to walk to the destination. A limitation I ran into is that Google’s Direction Service doesn’t support different transit methods for multiple waypoints. This, combined with my desire to get it out there, is why the design’s not as good as it should be. I’ll see if I can improve it over the next few weeks. People have also been telling me this needs to be on mobile so I’m going to use this as an excuse to jump into mobile development. I’m excited.


Citibike Station to Station Directions
Sat, 01 Jun 2013 00:00:00 +0000

 
 Photo by @rafat

On Wednesday, I took my first bike ride using New York City’s new Citibike program. So far it’s been great but one issue I ran into is being able to plan a trip. Google offers cycling directions from place to place but doesn’t take into account the Citibike stations. On the other hand, the Citibke app shows the rental stations but doesn’t make it easy to find directions from one station to another unless you’re already at one of them.


Adding attachments to django-postman
Fri, 17 May 2013 00:00:00 +0000
After doing a round of customer development for Makers Alley, we discovered that customers really wanted to communicate with makers about their pieces. In true MVP fashion, we got the first iteration out in a day by using django-postman to handle the user to user communication. Within a few days, we quickly discovered that text messages weren’t enough and we needed to support file attachments, otherwise makers can’t easily show their designs and customers can’t share what they like. Unfortunately, django-postman does not support attachments and we didn’t want to have to incorporate another messaging library. Another constraint was that we were already using the awesome jQuery File Upload library (in truth, a modified Django version by Sigurd Gartmann) to allow makers to upload images when managing their storefronts.


Eighteen Months of Django: Part 2
Fri, 10 May 2013 00:00:00 +0000
On Tuesday, I shared some best practices I picked up while using Django. This is a follow up post to share the packages that I found useful as well as various hiccups I encountered when using them.

 
 django-registration and django-social-auth: Combined, these packages let you handle the basic user registration and activation. Most likely, you will end up having to customize them a bit to do what you want. For example, allowing a user to register using an email address instead of a username or requiring an email address for a user who signs up using Twitter. A small issue that annoyed me is that the signals generated by these two packages occur at different points: django-registration generates signals that includes the request while django-social-auth generates signals that contain the response from the OAuth provider. Depending on your use-case, it may be worth it to use the simple backend for django-registration, it automatically activates and logs-in the newly registered users, making your app a bit easy to get into.
 django-storages and boto: If you plan on using S3 to host static content, definitely take a look at these. They provide backends to make it easy to save and access your static content to S3 without having to deal with the AWS API. I ran into some issues using this along with Cloudfront and django-compressor but I was able to fix them by looking at Stackoverflow.
 django-compressor: This is a neat library that will compress and minify your JS and CSS, check if anything’s been updated, generate an upload the result to static files location, and update the HTML to point to the new location. This makes sure that users never end up with older, cached versions of your static files. One thing to note is that you need to make sure that your Javascript are properly formatted and all end in a semi-colon; otherwise you run the risk of the compression failing. I know that there are other Django compressors out there but I’ve been happy with django-compressor.
 sorl-thumbnail and PIL: If you allow users to upload images this is a must have. It provides a standard way of resizing the images and caching the result. The library comes built in with support for cropping and a variety of other processing options so you don’t have to worry about it. One thing to note is that if a user is loading a page where none of the images have been generated yet, it will delay the page load until all of the images are generated. As long as you know the required sizes of all images, you can run a task on the backend to generate each of the images. You may have trouble installing PIL in a virtualenv but doing some Googling it should be easy to figure out.
 django-extensions: Just a neat library that comes with additional management commands to make developing Django easier.
 django-debug-toolbar: This intercepts every Django request and provides some debug information to help you optimize your code. The most useful piece to me is being able to see the SQL queries that are being executed and helps me figure out what needs tweaking/caching.
 django-crispy-forms: If you’re using Twitter Bootstrap, this is a library that lets you generate Bootstrap forms in Django.
 django-celery and celery: This is a way to run tasks in the background. With Pressi, we initially started with some management commands behind some cron jobs but we ended up switching to Celery when we wanted to distribute it across multiple machines and have built in support for threading and error handling. One thing to note is that we used RabbitMQ as the backend but it takes a bit of time to setup and I’m still struggling to understand the ways to manage it. A lot of people have been using Redis as the backend successfully and I think I’ll give that a go in future projects.
 mongoengine and pymongo: If you’re using Mongo, take a look at mongoengine, which serves as an ORM for Mongo, and is built on top of PyMongo, a Mongo API. Mongoengine makes it very easy to change your models from a relational database to an documented-based one by keeping the field types and model definitions similar. Be aware that document-based databases are significantly different from relational ones and that although cosmetically your models look similar, the interaction with the backend is very different. You shouldn’t switch to MongoDB just because you can - make sure you’re switching for the right reasons. For Pressi, we use a hybrid approach where we use MongoDB to store a user’s social media content with everything else stored in MySQL. Something to be cautious of is that both of these libraries have been evolving pretty quickly and we ran into an issue where we weren't able to consistently connect to a MongoDB instance until we stumbled unto the right versions of the libraries (in our case, 0.6.20 for mongoengine and 2.4.1 for pymongo).
 django-haystack: When you’re ready to graduate from implementing a search using QuerySet filters to an indexing backend, take a look at Haystack. It provides a pretty simple search interface that integrates pretty well with Django and supports a few different backends. We ended up settling on the Xapian backend because it was supposedly simpler but ran into some trouble installing it inside a virtualenv until I found this post. Note that although Haystack supports multiple backends, not all features are supported by every backend so make sure the backend you choose supports everything you need. I believe Solr has the most functionality out of the box but we wanted to keep it simple for Makers Alley.
 django-postman: We just implemented this for Makers Alley but it’s a very simple way of doing user to user messaging. It comes with the standard messaging features (inbox, reply, archive, delete) but one thing I wish it had was a way to include attachments.
 Fabric: I mentioned this in the previous post but wanted to reiterate it since it makes building and deploying your code easy. It also forces you to think about your environment and you end up with a better structured project as a result.
 South: Another package I mentioned earlier that makes it significantly easier to deal with database migrations in Django. The only time we've run into issues using South is when two of us were making changes to the same model in parallel branches. Even then it's easy to replace the two flawed migrations with a functional one.
 Unidecode: This isn’t a Django specific library but we found it useful when cleaning up unicode data. If you ever get random unicode exceptions in your code, Unidecode should be able to help.
 BeautifulSoup and PyQuery: If you need to do some HTML scraping in Python, take a look at BeautifulSoup. It turns HTML code into an object that’s easy to navigate and search. After getting more and more familiar with jQuery, I found a python alternative in PyQuery but am still getting comfortable with it. If you come from the jQuery world I’d try using PyQuery first; otherwise I’d try BeautifulSoup.
 requests: Just a nice and simple replacement of urllib and urllib2 that makes it much simpler to make HTTP requests. Your code becomes cleaner, more readable, and more expressive.

I tried to highlight the libraries that have made developing in Django easier but I’m sure there are tons more. I’d love to hear about them so do share.


Eighteen months of Django
Tue, 07 May 2013 00:00:00 +0000
I’ve discovered that every new project lets me correct mistakes from my earlier attempts by allowing me to start from scratch. This is especially true with a web framework such as Django that has a ton of little nooks and crannies that take a while to explore and understand. It’s usually not worth it to go back and fix something that’s not broken on a functional product but starting a new project lets me do it right from the beginning. Now that I’ve developed and launched (with Sandy and Marc) two serious Django-based products as well as bunch of smaller ones, I wanted to document some personal best practices I’ve picked up. Obviously, I’m still learning and I may be completely wrong with them so let me know if you disagree. If you’re interested in a deeper look at some of the topics let me know and I can write up another post going into detail about a particular topic.


Raspbmc
Wed, 10 Apr 2013 00:00:00 +0000

 

I’ve been interested in the Raspberry Pi ever since I first saw it mentioned in the tech news and finally got to play with it over the past few days when my brother (thanks Simon!) lent me an extra one he had. I’ve been in need of a better media center setup ever since my DisplayPort cable stopped working so I decided to try out Raspbmc, a Raspberry Pi based media center.


Mmmm... pseudo static sites
Tue, 12 Mar 2013 00:00:00 +0000
Reading Katie Zhu’s post on NPR’s news app architecture got me curious about a setup where most of the content is static and can be hosted on S3 and EC2 is primarily used to generate the static content which is then uploaded to S3. The benefits were obvious:

Cost: S3 is cheaper than EC2.
Reliable: S3 doesn’t go down near as frequently as EC2.
Scalable: Since it’s primarily static you don’t have to worry about additional capacity or dealing with caching, databases, and all the other fun things.
Simpler: There are no weird server issues here. As long as you generate the right content and your rendering is good, you don’t need to worry about a web server acting up.

I’ve been meaning to write a script that would scrape Hacker News in order to show me the top content I missed while sleeping. I had some time this weekend and decided to give it a go using this “pseudo-static” approach. The result is called Yet Another Hacker News Reader (YAHNR) and you can take a look at the code on GitHub. Turns out it was pretty simple to write and the most difficult part was thinking differently about the problem. Whereas I’d keep the content in a database I ended up storing them in static JSON files and instead of having the logic to generate the HTML page live on a web server I have it using Mustache templates.


Identifying duplicate bills across states
Tue, 05 Feb 2013 00:00:00 +0000
This past weekend I participated in the Bicoastal Datafest hackathon that brought together journalists and hackers with the goal of analyzing money’s influence in politics. I came in with the idea of analyzing the evolution of a bill in order to see which politician made the various changes and relate that to campaign contributions. I quickly discovered that that wouldn’t be very easy, especially in two days, but I did meet Llewellyn, a journalist/hacker, who had a more practical idea of programmatically identifying bills across states that used the same language. The intuition behind this being that it would identify bills that were unlikely to have been written independently of one another and likely to have been influenced by a 3rd party.


Making sense of my Twitter archive
Sat, 19 Jan 2013 00:00:00 +0000
I finally got access to my Twitter archive and decided to have some fun with it and also give me an excluse to play around with matplotlib. The first step was just seeing what the data looked like and what information was available. Turns out that Twitter included a simple HTML page to let you browse your tweets but also provided CSV files for each month. The fields were pretty self explanatory but one “gotcha” was needing to convert the timestamp to my local time. I wanted to do a few data visualizations to see what my tweeting behavior was like and also see if anything insightful came out. As I started looking at the visualizations I noticed that I’m more active than I used to be and that I have a pretty stable relationship betweet my tweets, my RTs, and my replies. In the future, I’d like to explore how my usage of Twitter has evolved and also get to play around with the NLTK library.


Web scraping like a pro
Wed, 09 Jan 2013 00:00:00 +0000
I’ve done my fair share of scraping ever since I started coding and just wanted to share some tips I’ve picked up along the way. I think scraping is a great, practical way to get into coding that is also immediately useful. It also forces you to understand the HTML of a page which gives you a great foundation when you’re ready to create your own site.
Hope they’re useful!


Self hosted Instagram export
Wed, 19 Dec 2012 00:00:00 +0000
I just hacked together a quick app to help download Instagram photos. At first, I tried using Instaport and OpenPhoto but both of them were backed up with others trying to do the same so I decided to create my own. It’s basically a really simple python web app that allows you do a quick authentication with Instagram and then lets you downloads all your images to your hard drive.


Trend of actor vs actress age differences
Wed, 23 May 2012 00:00:00 +0000
I recently watched Miss Representation which documents how the portrayal of women in the media affects women’s roles in society. It raised many interesting points and definitely got me thinking. If you haven’t seen it already you should definitely check it out. One of the points was that there’s a huge pressure to cast female roles with young actresses whereas it doesn’t matter so much for the male. I was sure this was true but I wanted to see how big of a deal it actually was, take a coding break, and play around with some data. The goal was to replicate the results as well as provide some tools for others to do similar analyses.

I took a quick look at the IMDB site and realized that they did not have an API available. I looked at a few open source alternatives but they all seemed like overkill for what I wanted to do so I decided to just write a quick Python script to scrape the pages I needed. I started by pulling the top 50 movies for each decade (via http://www.imdb.com/chart/1910s - http://www.imdb.com/chart/2010s) and then pulling the top 5 cast members for each movie (via http://www.imdb.com/title/tt1375666/fullcredits#cast). I had to actually look at the actor/actress pages as well in order to pull the birth dates as well as the sex. After loading this data into a database it was a very simple query to run the analysis and then Google Spreadsheets to clean it up. 

Not surprisingly, it turns out that over the past 11 decades, the average actor is 41 while the average actress is 32. Interestingly, during the 1980s they were almost the same but the gap has been widening since then.


Tech interview question
Sat, 08 Jan 2011 00:00:00 +0000
When conducting interviews, I’ve developed the following criteria for a good interview problem:

	Avoid brain teasers - they tend to be hit/miss and some people don't really do well under this type of problem
	Challenging - the answer should not be immediately obvious and the should require some creativity
	Rare - similar to above, the problem should not be a common question in order to get
	Flexible - the problem has multiple solutions and can be modified on the fly for different skill levels

I’ve found that the following problem satisfies the criteria and gives a pretty good sense of a developer’s skill level.


Enabling modules in Apache2 under Ubuntu
Fri, 23 Jan 2009 00:00:00 +0000
The Apache enabled modules are found in "/etc/apache2/mods-enabled" as a set of .load and .conf files. If the modules you want are in the /etc/apache2/mods-available folder but not in "/etc/apache2/mods-enabled" folder, just copy the .load and .conf files over (note that the .conf file may not exist).
If there is no file in the mods-availble folder, you will need to create a new .load file in the mods-available folder to point to a module in "/usr/lib/apache2/modules". To do this, create a .load file containing the line "LoadModule xxx /usr/lib/apache2/modules/yyy.so" where xxx is the name of the module and yyy is the file name. After creating this file, you can just copy it over to the mods-enabled folder and restart apache using "sudo /etc/init.d/apache2 restart".


Interesting Perl behavior
Fri, 30 May 2008 00:00:00 +0000
I ran into this problem a while back and wanted to share it. It was a bit unintuitive but documentd so I guess I shouldn't be surprised by the results. Hopefully this will help someone else avoid this pitfall.
It looks as if declaring a variable with the "my" statement but then guarded with an "if" statement causes the scope of the variable to be global - note that the "use strict 'vars';" pragma does not give an error in this case.