Ryan McGrath is an Entrepreneur, Engineer, Designer, Author, and Teacher. Buzzwords, eat your heart out.

Ramblings, code, and more. Find me around the world for free coffee.

Tue 22 March 2016

A Deep Dive into PL/v8

Back in August, Compose.io announced the addition of JavaScript as an internal language for all new PostgreSQL deployments. This was thanks to the PL/v8 project, which straps Google's rocket of a JavaScript engine (V8) to PostgreSQL. This got me thinking - PL/v8 offers a rich and featureful way to write functions in PostgreSQL. Let's take a look at what it offers by building a mini JavaScript module system on top of it, complete with basic support for the CommonJS API.

Enabling PL/v8 on Your Deployment

First thing's first: you'll need to enable it by creating the extension. The quickest and easiest way to do this is likely using psql from a terminal (below), but if you prefer using another tool it shouldn't pose any problems.

PL/v8 also supports two extra JavaScript "dialects" - CoffeeScript, and LiveScript. CoffeeScript offers a more Ruby-esque syntax, and LiveScript is a more functional successor to CoffeeScript. We'll be using pure JavaScript in this article, but if you'd like to use either of these languages you'll need to create their extensions below as well.

JavaScript in PostgreSQL: The Basics

Creating a function using PL/v8 looks like any other PostgreSQL function, with the exception of a language specifier change. Take the (basic) example below: we're simply incrementing each int in an Array by 2, and returning it as pure JSON.

JavaScript isn't technically a functional programming language, but it has many elements of one. Those features shine here - mapping over results is incredibly expressive and easy to work with. While you can't get all crazy with any ES6 code yet (the version of V8 currently used is a bit older), pretty much all the good parts of ES5 are up for grabs. Couple that with the native JSON support PostgreSQL offers, and you can get many of the features (e.g, Schema-less storage) from Document-oriented databases like MongoDB or RethinkDB.

As far as working with SQL data in JavaScript goes, it's relatively straightforward. PL/v8 will convert most database types automatically for you, including polymorphic types (anyelement, anyarray, anyenum and anynonarray) and bytea (through JavaScript's Typed Arrays). The only real "gotchas" to be aware of are that any Array you return must be flattened (unless you're returning JSON, in which case go for it), and you can't create your own Typed Arrays for use in functions; you can, however, modify and return the ones passed to your function.

While not strictly a "gotcha", the ever-so-fun issue of context in JavaScript is present in PL/v8. Each SQL function is called with a different this value, and it can be confusing at first. SQL functions do share context though, as far as accessing globally declared variables and functions. As long as you pay attention to scoping issues, you can avoid context binding issues and write reusable code.

Hacking Together a Module System

V8 is synonymous with Node.js for many developers, and inevitably the question of importing modules comes up. There is no baked-in module system, but we can simulate one using some of the features of PL/v8. It's important to note that while this works, we're in a sandboxed environment - modules involving network calls or browser-related functionality won't work. We'll be simulating the CommonJS module.exports API though, so many modules should "just work" right off npm.

The first thing we'll need is a table to store our module source(s) in. We really only need two columns to start: the module name (module), and the source code (source). To sweeten the deal we'll add an autoload column (autoload) that we'll use to dictate whether a module should be transparently available at all times.

We'll need a function to handle wrapping the require() API, and ideally we'll want a cache for module loading so we're not pulling from a database table every time we require a module. The global plv8 object has a few things we'll make use of here - it brings important functionality like executing statements, subtransactions, logging and more to the table. We'll be eval()ing the source for each module, but we wrap it in a function to ensure nothing leaks into the global scope. Our autoloading of certain modules also takes place in this function, just to prime the module cache for later use.

Now in terms of using this, we have that dangling context problem to consider - how do we make sure that require() is available to each PL/v8 function that needs it? Well, it just so happens that PL/v8 supports setting a specific function to run before any other functions run. We can use this hook to bootstrap our environment - while ordinarily you could set it in your config files, you don't have access to those on Compose. We can, however, SET this value every time we open a connection. As long as you do this prior to any function call (including CREATE FUNCTION itself) you'll have access to require().

Let's try it out by throwing together a module that implements the Fisher-Yates shuffle algorithm - we'll name the module "shuffle", to keep things simple, and go ahead and set it to autoload.

Now we should be able to require() this! We can try it immediately - a simple table of people and a super readable random_person() function works well.

See how clean that becomes? A shuffle algorithm is only one application - modules like lodash are a prime candidate to check out for using here.

What Are You Waiting For?

Writing PostgreSQL functions in JavaScript can provide a few wins - if your stack is primarily JavaScript, there's less context-switching involved when dealing with your database. You can reap the benefits of Document-oriented, Schema-less databases while leaving yourself the option to get relational as necessary, and the native support for JSON marries perfectly with JavaScript. Modules can tie it all together and provide a method for organizing code that makes sense - you'll feel right at home.

Even more in-depth documentation on PL/v8 can be found over on the official docs. Try it out today!

Thu 04 June 2015

RedNoise, a Django-centric WhiteNoise addon

I've developed with Django for a number of years - out of all the frameworks I've ever used, it strikes the best balance between "let me get stuff done" and "don't try to provide me too much, get out of my way when I say so". I was ecstatic when WhiteNoise was released, as it solved a very annoying part of the development process with Django - static files. Rather than uploading things to S3 (ala the old Storages route) it's now just easier to toss CloudFront in front of your files and let uwsgi serve it up. Since WhiteNoise debuted, I've used it on a few different projects, and over time I found myself wanting a few things that it lacked - django-rednoise provides them.

Feel free to read on for details, or go find it on GitHub! I published it as a separate module as the goals differ from the established WhiteNoise goals, but I'd be absolutely open to it being merged or pulled from.

DEBUG Please

WhiteNoise has a Django-specific module that you can use, but it's essentially geared towards production use-cases. I tend to prefer having a development setup that mimics my production setup; patching Django urls to load static or media files in development just feels clunky to me.

RedNoise respects whether Django is in DEBUG mode or not. If DEBUG is set to True, RedNoise will mimic Django's default static-file loading pattern, so you don't need to reload the entire server just to debug some frontend issues.

Serve Media Files

WhiteNoise doesn't support serving user-uploaded media files, but I wound up having to throw together a CMS at one point and ran into this limitation. Content authors and editors wanted to be able to upload photos, but I didn't want to have to keep a separate S3 bucket for it all. RedNoise will serve media files, hooking into the associated settings.py parameters to make it "just work".

Should You Use It?

I've run it for a bit with no real issues; I would say that whatever you do, tossing a CDN in front of it all is pretty efficient. Provided you do that, the CDN should absorb most of the requests, leaving your server to do its thing.

If you use WhiteNoise, and wish you had the above features, give django-rednoise a shot. It's just a pip install away.

Wed 08 January 2014

Recording Live Audio Streams on iOS

Some people might consider this strange, but I'm still a fan of radio. It's not my 24/7 music source, but I find myself getting annoyed with algorithms trying to predict music that I'd like and ultimately failing. In addition to music, you've got news and sports radio - getting a large mix is fun.

These days tons of radio stations have moved their streams to be listen-able online, and recently I found myself wishing I had something that I could use to record these radio streams on a whim. My main device is an iPhone 5; sadly, I didn't find anything particularly pleasing or enjoyable to use in the App Store, so I set out to see about throwing together my own. What follows is a breakdown of a (relatively) easy way to download live audio streams using AVFoundation and Core Audio. It assumes you've got a working knowledge of Objective C, as well as a working knowledge in terms of building iOS apps in general.


This code is here as an example and nothing more. In recording anything from a live stream that might possibly be copyrighted and/or legally protected you should ensure that you're allowed to do so. This post and the author holds no responsibility for what you do with the content hereafter.

Update 2015

In the comments spg has shown a case wherein .m3u8 files probably won't work with this method due to how they differ in playback. It would be cool to see if there's a way around this, but I don't have time at the moment to investigate sadly. Get in touch if you figure this out!

Update 2016

I've sinced removed comments from my site due to their neglect. Feel free to email me with questions about this though!

Audio Streaming

The basics of app structure/building aside, the first concern is the actual streaming of the audio. There are a myriad of factors here that are honestly just a massive chore to deal with - you've got network speed issues, you could have the network drop and pick back up, the list is endless. A cursory Google search tends to show that this is where people tripped themselves up a lot - the general consensus seems to be that relying on AVPlayer can't work for remote data, as there's seemingly no way to get at the AudioBuffer data - it only works with local files. This leads to everyone reinventing the wheel with the (arguably painful) Core Audio APIs.

Here's the thing: I had no desire to deal with any of that. They are certainly interesting problems, but right now they're just in the way. If possible I would much rather use AVPlayer and let Apple handle all the intricacies/edge cases in regards to playback.

Enter MTAudioProcessingTap

Now, MTAudioProcessingTap is no secret - it's what you'd use to, say, visualize audio data for local files played through AVPlayer. If you're not familiar with it, this is a pretty good writeup. The general gist is that you create a tap, set up an audio mix for a track, and set it on the AVPlayerItem of the audio player. The problem with remote data is that AVPlayer just works differently there - you don't have access to any AVAssetTracks with which to make an AVMutableAudioMix, presumably because streaming is just a different setup entirely behind the scenes.

However, if we look a bit further, after the streaming starts, you can access an AVAssetTrack on the player. Now it's a simple matter of KVObserving the status of the player, grabbing the track when it's available, and setting up our stream handler. Given that it's MTAudioProcessingTap you could do any number of things here, but personally I just needed to pass the raw audio data through.

Unlike other articles on this site, I'm opting to just include three gists at the bottom that act as a full (mini) library, along with example usage. There's a bit of Core Audio involved, but it's nothing too annoying - hopefully this helps anyone who's been wondering how to handle this. This code isn't meant to be drop-in-good-to-go; with this type of project it's kind of expected that you integrate it based on your own needs.

If you have questions, you can always feel free to get in touch, be it email, Twitter, or GitHub. Generally down to help!

Wed 10 April 2013

We are the New Blue Collar

Over the past year, there's been an explosion of interest in terms of teaching children how to program in schools. This is a great movement; the world is becoming more digital by the day, and as we automate away jobs with technology we certainly need more people to help maintain that automation. While it's not a 1:1 replacement for the jobs that get lost in the process, it's an industry that's not going anywhere. Organizations such as Treehouse or Codecademy, and efforts such as code.org are helping educate others when it comes to the simple fact that these skills are useful in todays economy and cannot be ignored.

The thing is, as someone who's lived and worked "in the trenches" for the past seven to eight years, the rhetoric coming out of some of these movements is somewhat alarming. I think this is doing a major disservice to the future generations of programmers, and I wanted to take a minute to explain what learning to program really does for you, career-wise.

The Hidden Side of Programming

There's a few things these movements don't teach you about programming for a living.

  • Simply learning to program does not automatically open doors for you - being successful in programming requires drive and hard work just like any other industry.
  • A great deal of programming jobs are based in maintaining a code base, not "writing the future" or the latest amazing technology. Generally, we as programmers fear jobs that are just maintenance. It's not something that will be explained outside of the core of the industry - these jobs don't typically offer you advancement opportunities, they stifle your creativity and generally cause you to consider looking for a new job, often going against what can be (in many cases) a considerably decent salary.
  • That aforementioned "considerably decent" salary will be driven down over time as more and more programmers become available. What's posited as a great reason to learn to code right now (money) won't necessarily be the case in a few years.
  • Ultimately, what's often thrown around as you get further into this career is the fact that nobody wants to be doing this when they're 40 years old. At some point you either try to move up to management, or change careers (or retire, if you're lucky enough).

If you're following along, this isn't really considered a long-term career path. As the world becomes more and more digital, being able to program becomes the same kind of skill as being good with your hands and being able to follow some directions. Yes, knowing how to control the devices and technology around you enables you to "build out your dreams" or "own your destiny", but it's not a given - it requires levels of drive and dedication that some people aren't willing to put out there.

The entire world cannot be artists, or it wouldn't function. If I talk to any programmer I know who actually enjoys doing this for work, they consider themselves an artist first and a data wrangler second; it's not a particularly enjoyable field to be in just for the money. These educational programs and efforts sell themselves as a way to realize your dreams, and while that's a lofty goal, it's just not a given because you can now write some code.

Dissecting some Lofty Quotes

  • "Coding is the American Dream. If you want to be the next Mark Zuckerberg or even want a high-paying job, those jobs are for programmers. … And yet the opportunity to be exposed to that is going to the top 10 percent, and that is just morally wrong." Hadi Partovi, Founder, code.org

I would hesitate to refer to "Coding" as the "American Dream". The role of a programmer is easily equated with any service-industry level job, albeit using more of your brain than your hands. You're still beholden to someone else at the end of the day, someone who controls when you get paid, someone who expects more and more with less and less. Programming is not a safe haven from the economic woes we face today.

I believe the point made at the end of this quote to be very accurate, though: the top 10 percent shouldn't have the only access to essential knowledge deemed 'critical'. The thing is that this statement could be used for just about any industry out there today - Law, Medicine, even Craftsmanship; our society doesn't enable easy access to this information any more than it does with programming. It's a larger problem that needs to be solved.

  • "Coders change the world. They build new, amazing things faster than ever before. Anyone with imagination can learn to write code." Jeff Wilke, SVP Consumer Business, Amazon.com (via code.org)

This is an incredibly bland quote that glazes over the fact that most programming jobs don't actually entail building new amazing things. It is very, very possible that you will wind up maintaining someone else's code or editing a trainwreck that the person before you created. It's not all innovation and aspirations.

  • "We need people from all walks of life to learn to code. Control your destiny, help your family, your community and your country." Mitch Kapor, Founder, Lotus Development Corporation & Partner, Kapor Capital (via code.org)

Another bland quote that implies that being able to program helps you control your destiny. If you have a desire to learn, you're in control of your destiny. Subscribing to a single discipline doesn't guarantee you success or even (necessarily) the chance to obtain your goals. It's a tool, a utility.

  • "Our programmers are our artists. They write beautiful code that millions of our travelers start to play with, to use, to help them travel better. It's a profound sense of power and satisfaction to have your work affect so many, so quickly, across the globe." Dara Khosrowshahi, CEO, Expedia, Inc. (via code.org)

I've worked with a ton of programmers in the past, at different companies of varying sizes (startups, bigger companies, freelancing/contracting), all over the world, and as a result I don't feel off the mark with this statement: you wind up not caring about this two weeks in. As long as you're creating for someone else and some company owns what you build, you begin to back off of the idea of going all in. You stop being wow'd by the idea that millions of people use what you wrote - you realize it's the network effect, and you could go work any number of other computer-related jobs and have the same thing. This isn't to say you're a passion-lacking worker-bee; you might have aspirations of helping people around the world, and that's a good thing - I'm just illustrating what's not often shown.

You'll see many deny this, though, as the tech industry has a problem with speaking out, and has what could arguably could be considered a very Japanese work ethic - you live and die by the company you work for.

Granted, we're not at the point of literally dying, but the hours and expected commitment have many sad comparisons. It's also not to say you have to fit that mould - it's just noting an environment you could easily wind up in. You're expected to care 110%, or they'll find someone else who can do what you won't.

  • "If you can program a computer, you can achieve your dreams. A computer doesn't care about your family background, your gender, just that you know how to code. But we're only teaching it in a small handful of schools, why?" Dick Costolo, CEO, Twitter (via code.org)

Whether a computer cares about you or not is totally irrelevant in terms of whether you should learn how to program or not. We should be teaching it in more schools, but we should be teaching it for the right reasons - it's essential, and can be an art, but it is not some almighty savior.

  • "The extensive Treehouse library of step-by-step video courses and training exercises will give you a wide range of competitive, in-demand technology skills that will help you land your next dream job or build your startup idea" Treehouse.com
  • "Our mission is to bring affordable Technology education to people everywhere, in order to help them achieve their dreams and change the world." Treehouse.com

The company that's going to get involved with schools and teach children how to program... is presenting themselves as a way to land a dream job or become your own boss. The former isn't always the case, and the latter is in actuality such a rare occurrence that most people never have it. I greatly admire them for reducing the costs surrounding teaching this material, but positioning this as a world changing proposition for the people who choose to give it a shot is ultimately misleading.

So Should I Learn to Code?

Yes. A million times yes. Just do it for yourself; don't do it for the money (because it won't always be there), and don't do it because people tell you it's job security - you can't possibly do it forever, and unless you plan your retirement correctly, good luck when you're 40. If you don't get into it as an art, then you're getting into nothing but new-age labor. You become a mechanic of the digital age.

I believe programming should be taught in schools, and I 100% agree it's an essential skill in the world today. I do not, however, agree with the people who seem to believe that it's the most life changing thing you could do for yourself (of course, I encourage you to determine this for yourself). If we're not careful, we're going to do to the next generation what earlier ones did to us. We have a country of college graduates who were brought up on the idea that there'd be jobs for them, because it's what they were led to believe from those who came before them.

They were wrong.

Teaching programming is an important goal as we move forward, but it is not a fix for a broken educational system. If we want kids to succeed, we need to continue to focus on fixing that - rooting out the bad teachers and paying teachers more. We need to fix the problem in this country that people are going hundreds of thousands of dollars in debt just to have a decent paying job.

The aforementioned programs don't fix these problems; they're simply training for the newest blue collar job.

Note: code.org is not full of bad quotes; I think it's a genuinely good idea as far as movements go. The quotes I pulled are indicative of the problem, a case of making the field out to be something it's not.

Looking for More?

I've been writing for quite some time! You may want to check out the Essays section for a full list.