Sql on Dan Goldin

Analyzing AWS ELB logs

Tue, 20 Feb 2018 00:00:00 +0000

Logging HTTP requests should be enabled for every application you run. When things go wrong, and they will, it’s often the first step to understand the problem. Unfortunately, logging isn’t always top of mind and is often forgotten. Luckily, if you use the Elastic Load Balancer (ELB) functionality within AWS you’re able to set up ELB logs that track every request and write it to an S3 bucket. The documentation is up on the Amazon site but there’s a surprising amount of information that’s hidden away in the logs. Since it’s Amazon and they want to make it as easy for you to use their various services together it’s simple to load the logs into Redshift and start digging into them via some basic queries.

Analyzing IMDB data: Step 1 - Cleaning and QA

Sat, 21 May 2016 00:00:00 +0000

In 2012 I did a simple analysis of IMDB to analyze the change in actor and actresses’s ages over time. At that point I limited the analysis to the top 50 movies each decade and hacked together a quick script to crawl and scrape the IMDB analysis. A couple of weeks ago I came across a great post by CuriousGnu that did a similar analysis across a larger set of movies but limited to movies since 2000. I reached out and they were kind enough to give me a DigitalOcean instance containing the data already loaded into MySQL. The analysis should be finished up tomorrow but I wanted to write this post up to share the mundane parts of the process. The janitorial part is critically important to an analysis and it’s important to get it right or the results will may be meaningless or even completely wrong. The NY Times interviewed a variety of data scientists and came away with the conclusion that 50 to 80 percent of a data scientist’s time is spent cleaning the data. This is no exception and I wanted to provide a sense of the effort and thought that goes into getting data into a state that’s actually useful.

Identifying unused database tables

Wed, 11 May 2016 00:00:00 +0000

When writing code it’s very easy to accumulate deprecated database tables that end up as zombies - they’re still around and may even be populated and used by a variety of side scripts but if they disappeared and the dependent code was removed nothing would be different. In fact you’d have a smaller code base, a smaller database, and would hopefully improve everyone’s productivity a tiny bit.

Dealing with the tables are are still being populated and read requires a bit of investigative work and knowledge of the product since there’s no simple way of identifying them. But there are a simple ways to identify tables that are no longer updated.

The MySQL enum type

Thu, 10 Mar 2016 00:00:00 +0000

The MySQL enum field provides a nice compromise - the space efficiency of using an integer, the human readability of text, and basic type safety. Yet I had this vague recollection of reading something that made it seem enums carried a ton of risks when changing the column definition so wanted to see if I could “break” it. Turns out it’s a lot more resilient than I thought. I went through a series of combinations - ranging from changing the order of the enums in the definition to trying to insert values that didn’t exist but in every case it handled it as expected. Doing a bit of research I discovered how MySQL represents the enum type. Rather than storing the values in a specific order MySQL supposedly creates a map-like structure to relate the integer values with their enum counterparts. This allows you to change the order of the enum definition without changing the underlying map or any of the stored values. I still wouldn’t use enums for anything that would require a join but for storing small and simple sets of data it works great.

More MySQL fun

Sat, 21 Nov 2015 00:00:00 +0000

I had a bit of fun with MySQL earlier this week when trying to explain a non obvious “group by” behavior. It’s fairly common to want to manipulate a field in order to transform it into something more useful. The difficulty arises when you want to keep the original name. Below is some SQL code that highlights the odd behavior.

drop table if exists dan_test;

create table dan_test (
 id int not null,
 id2 int not null
);

insert into dan_test (id, id2) values (1,1), (2,2), (3,3);

select * from dan_test;

select id, case when id = 1 then 2 else id end as id, id2
from dan_test;

select id, sum(id2)
from dan_test
group by id;

select case when id = 1 then 2 else id end as id, sum(id2)
from dan_test
group by id;

select case when id = 1 then 2 else id end as new_id, sum(id2)
from dan_test
group by new_id;

Comparing SQL schemas
Sun, 12 Jul 2015 00:00:00 +0000
During development it’s common to get your dev database out of sync with the one in production. Sometimes it’s due to an additional column in development you added before realizing it wasn’t necessary and other times it’s just creating a few temporary tables on production that you forget to drop. In both cases it’s useful to reconcile the schema differences every once in a while to keep your database in a clean state. In the past I would just run a simple query (select table_schema, table_name, column_name from information_schema.columns;) on each environment and then use either Excel or Google Sheets to spot the differences. This takes a bit of time so this weekend I put together a quick JavaScript tool to automate the process. You simply run the schema query on each of the environments and paste the resulting rows into the two text areas. The result is a JSON based diff showing the additions, deletions, and modifications to each of the tables and fields. The next step is to modify it to also identify differences in the column types.


Ambiguous SQL queries
Sat, 27 Jun 2015 00:00:00 +0000
One of the best habits to develop when working with SQL is to always refer to fields through an alias. Numerous times I decided to just take a shortcut and ended up regretting it later. Even if you’ve tested your query to make sure it works there’s no guarantee that a future change to a table schema won’t break it.
Let’s say you have the following two tables - with items.category_id corresponding to categories.id


A MySQL “GROUP BY” nuance
Tue, 09 Jun 2015 00:00:00 +0000
I discovered a nuance with MySQL’s GROUP BY statement earlier today that I’ll share with the hope that others can learn from it. It’s fairly common to use a coalesce statement to handle null values while keeping the resulting field the same name. For example:
SELECT coalesce(a.user_id, b.other_user_id) as user_id, sum(s.num) as total_nums
FROM table_a a
LEFT JOIN table_b on a.some_id = b.some_other_id
LEFT JOIN stats s on a.stat_id = s.id
GROUP BY user_id;

MySQL vs PostgreSQL sort order
Sun, 01 Feb 2015 00:00:00 +0000
At TripleLift, we have a migrations job that copies aggregate data from Redshift to MySQL so it can be accessed along the rest of the transactional data. As part of a test, I tried comparing that the data matched exactly but ran into an issue when exporting the data to select. Namely, to make the comparison as simple as possible I wanted to run the same select query in both tables and compare the results. Unfortunately, the sort order between MySQL and PostgreSQL (what Redshift is based on) acts differently for text fields. PostgreSQL takes case into account while MySQL does not. This has an especially weird results when you have values that contain characters with an ASCII code between the lower and upper case letters: []^-`. It took some research but I discovered that MySQL provides an option to do a case sensitive sort - just add a “BINARY” option before the field name.


Normalizing a CSV file using MySQL
Wed, 01 Oct 2014 00:00:00 +0000
As part of my preparation for the Intro to MySQL class I decided to put together a dataset we’d be able to explore over the course of the class. While trying to think of an interesting dataset to use I remembered I had a script that scraped Yahoo’s fantasy football projections for the 2014 seasons that I used to prepare for my draft. The only issue was that the script generated a CSV file so I had to go through a series of steps to turn it into a clean, relational database. I thought it would be useful to share the commands below and provide some context for those interested in learning more about MySQL and the data import/cleanup process.


The future of databases
Sat, 05 Jul 2014 00:00:00 +0000
A couple of weeks ago I attended a talk by Professor Michael Stonebraker. For those unfamiliar with him, he’s a database researcher responsible for PostgreSQL, Vertica, VoltDB and a dozen others. During his talk he shared his thoughts about the future of databases and what we can expect to see in the coming years. His main point is that databases are becoming more and more specialized and it will be very common for companies to run multiple types of databases that are optimized for different uses cases.


Solving coding tests in PostgreSQL
Sat, 25 Jan 2014 00:00:00 +0000
Most developers are familiar with the FizzBuzz code test which is a quick way to filter out developers who can’t code. At Yodle, we had our own, slightly more challenging problem. The challenge was read in a text file and then print out the frequency each word appears in descending order. It’s more complicated than FizzBuzz but it assesses a variety of skills. The solution needs to do the following:


Using the information_schema.columns table
Sun, 15 Dec 2013 00:00:00 +0000
Something that’s been really helpful to me in understanding a MySQL database is the built in information_schema.columns table. It provides information on every column in the database and is queryable just like any other table. This makes it easy to quickly find all tables that have a particular column name or all columns that are the same data type. There have been countless times where I knew the data existed somewhere but couldn’t recall which table it was in. Querying the information_schema.columns table for the foreign key helped me quickly figure it out. Below are some sample queries that retrieve data from the information_schema.columns table:


RDS and R
Fri, 15 Nov 2013 00:00:00 +0000
In my quest to replace Excel with R I’ve been spending the past week trying to do everything in R. It hasn’t been that easy with many things taking longer due to me having to reference the R docs but one thing that’s been great so far is being able to quickly run a query on Amazon’s RDS and pull data into a data frame for quick analysis. Being able to wrap this into a reusable function makes things even better. The one thing that makes it tricky was not being able to connect to RDS directly but having to tunnel through an EC2 instance. Below are the steps to replicate the setup.