Messing with PostgreSQL users and permissions

So we needed to add a Postgres user that could only read tables from our database--not only the currently existing tables, but any tables added in the future--and I found Postgres's permissions system surprisingly complex and unfriendly.

Sure, everything you need to know is buried throughout multiple manual pages and tons of Stack Overflow answers, among other places, but I didn't feel there was anything out there that brought together all the disparate sources into one easy to follow beginner's guide to understanding Postgres permissions, so here we are, the easy guide to creating permanent read-only users in PostgreSQL1.

Before we dive in, I'm using version 9.3 of Postgres, on Ubuntu 14.04:

$ psql --version
psql (PostgreSQL) 9.3.11

So all this information will apply to that version of Postgres, and all the commands will work on that version of Ubuntu. I'm not sure what changes, if any, would need to be made for earlier, later, or different versions of Postgres and Linux.

Structure

It really helps to understand some of the different data structures that come into play when messing with Postgres's permissions, by default, every database you create has a default schema named public, the schema you use is incredibly important and could be a great source of frustration and annoyance2 as you mess with the users and try and set their permissions.

But it won't be frustrating for us because we're going to learn by example, on a fresh install of Postgres, let's create our two users, our owner foo and our read only user readonly:

$ sudo -u postgres psql -c "CREATE USER foo WITH ENCRYPTED PASSWORD '...'" -d template1
$ sudo -u postgres psql -c "CREATE USER readonly WITH ENCRYPTED PASSWORD '...'" -d template1

and let's have our new foo user own our two databases:

$ sudo -u postgres createdb -E UTF8 --locale=en_US.UTF-8 -O foo db1
$ sudo -u postgres createdb -E UTF8 --locale=en_US.UTF-8 -O foo db2

Now, let's create a table in db1:

$ psql -U foo db1
db1=> CREATE TABLE bar1 (_id INT);
CREATE TABLE
Time: 8.384 ms
db1=> \q

and a different table in db2:

$ psql -U foo db2
db2=> CREATE TABLE bar2 (_id INT);
CREATE TABLE
Time: 9.042 ms
db1=> \q

So now database db1 should have a bar1 table, and database db2 should have a bar2 table. Now we're ready to understand the public schema of the database.

What we are going to do is connect to db1 and list its tables, then connect to db2 and list its tables:

vagrant@vagrant:~$ psql -U foo db1
db1=> \d
List of relations
-[ RECORD 1 ]--
Schema | public
Name   | bar1
Type   | table
Owner  | foo

db1=> \c db2
You are now connected to database "db2" as user "foo".
db2=> \d
List of relations
-[ RECORD 1 ]--
Schema | public
Name   | bar2
Type   | table
Owner  | foo

Notice that each table belongs to schema public, but each public schema has a different table. So now we understand that when we mess with the public schema, we need to be cognizant of what database we are connected to because any changes we make to the public schema will only affect the public schema of the connected database.

The second thing we should undertand is Postgres seems to use role, group, and user interchangeably:

The concept of roles subsumes the concepts of "users" and "groups". In PostgreSQL versions before 8.1, users and groups were distinct kinds of entities, but now there are only roles. Any role can act as a user, a group, or both.

We're only ever going to use user here, but when looking at different sources of information, you might find people using role or, less likely, group3.

The readonly user

We want the readonly user to be able to read all tables from db1. Before we do anything, let's see where we stand:

$ psql -U readonly db1
db1=> select * from bar1;
ERROR:  permission denied for relation bar1
Time: 1.043 ms

One of the first things you noticed (hopefully) is our readonly user could connect to db1 without doing anything, this is because users have CONNECT privileges by default, but even though readonly could connect to db1, we couldn't look at the table because we hadn't given readonly any privileges to do so.

OK, so we better dig into the manual and learn all about the GRANT query, and since we've already spent some time learning about schemas previously, we understand each of our databases has a separate public schema, and those schemas contain our tables, so we probably want to grant some permissions to readonly in our public schema, something like:

$ sudo -u postgres psql -d db1 -c "GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly"
$ sudo -u postgres psql -d db1 -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly"

Let's test it out:

$ psql -U readonly db1
db1=> select * from bar1;
(No rows)
Time: 0.758 ms

db1=> \c db2
You are now connected to database "db2" as user "readonly".
db2=> select * from bar2;
ERROR:  permission denied for relation bar2
Time: 0.939 ms

It looks like it worked, but just to be sure, let's double check, first, we'll try and create a new table:

$ psql -U readonly db1
db1=> CREATE TABLE bar3 (_id INT);
CREATE TABLE
Time: 6.279 ms
db1=>

Oh snap! That shouldn't have worked, what the heck? Well, let's dig into the permissions of the public schema:

$ psql -U readonly db1
db1=> \dn+
List of schemas
-[ RECORD 1 ]-----+-----------------------
Name              | public
Owner             | postgres
Access privileges | postgres=UC/postgres
                  | =UC/postgres
Description       | standard public schema

What does that =UC/postgres mean? I'm glad you asked, the answer can be found, of course, in the manual:

rolename=xxxx -- privileges granted to a role
        =xxxx -- privileges granted to PUBLIC

            U -- USAGE
            C -- CREATE

        /yyyy -- role that granted this privilege

Huh, the public schema has CREATE privileges by default, well that was unexpected. So we need to remove those privileges from the readonly user, should be easy enough using a revoke query:

$ sudo -u postgres psql db1
psql (9.3.14)
Type "help" for help.

db1=# REVOKE CREATE ON SCHEMA public FROM readonly;
REVOKE
Time: 3.782 ms
db1=#

Easy peasy, let's try again:

$ psql -U readonly db1
db1=> CREATE TABLE bar4 (_id INT);
CREATE TABLE
Time: 7.259 ms

WTF?

wtf animated gif
wtf?

Turns out, revoking a privilege from the user directly doesn't override the granted privileges on public. Why? I don't know:

@user3669651: But it's not the same. Revoking privileges from readonly2 doesn't change the fact that every user can create tables in the public schema until you revoke from role public ... – Erwin Brandstetter Jun 20 '14 at 23:32

It really doesn't make any sense to me, but this is the reality we have to deal with, and we need our readonly user to not be able to create tables, so we have to change up the public schema:

$ sudo -u postgres psql -d db1 -c "REVOKE CREATE ON SCHEMA public FROM public"

And, of course, we'll want to give all the permissions back to our owner, because if we don't, they won't be able to create tables anymore (go ahead, ask me the fun way I discovered this):

$ sudo -u postgres psql -d db1 -c "GRANT ALL ON schema public TO foo"

Now, our readonly user shouldn't be able to create tables anymore:

$ psql -U readonly db1
db1=> CREATE TABLE bar5 (_id INT);
ERROR:  permission denied for schema public
Time: 1.082 ms

Boom, we're finally making progress! I'm sure it's just a formality, but let's make sure we can read new tables also:

$ psql -U foo db1
db1=> CREATE TABLE bar5 (_id INT);
CREATE TABLE
Time: 7.252 ms
db1=> \q
$ psql -U readonly db1
db1=> select * from bar5;
ERROR:  permission denied for relation bar5
Time: 0.950 ms

And we were doing so well.

doing-so-well animated gif
We were doing so well, and then...

So now we'll need to spend some more time researching, where we'll eventually realize that granting permissions only works for things that existed the moment those permissions were granted, but if we wanted to have those permissions moving forward for all the new things, we'll need to alter the default privileges that are applied when those new things are created, so let's do that:

$ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT, USAGE ON SEQUENCES TO readonly"
$ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT ON TABLES TO readonly"

These are very similar to our original GRANT queries (which feels like we executed a lifetime ago now).

Let's make sure the queries worked:

$ psql -U foo db1
db1=> CREATE TABLE bar6 (_id INT);
CREATE TABLE
Time: 5.025 ms
db1=> \q
$ psql -U readonly db1
db1=> select * from bar6;
(No rows)
Time: 0.816 ms

Well what do you know, success!

celebrate animated gif
it's time to party

The most important part of those ALTER DEFAULT commands is the ...FOR USER foo..., if you don't have that part they won't work (which would lead to quite a while trying to figure out why the query does nothing, not that I would know), and it's also important to run those queries for any user that is going to be creating stuff in that database, because:

$ sudo -u postgres psql db1
db1=# CREATE TABLE bar7 (_id INT);
CREATE TABLE
Time: 5.430 ms
db1=# \q
$ psql -U readonly db1
db1=> select * from bar7;
ERROR:  permission denied for relation bar7
Time: 0.920 ms
db1=> \q

So you would need to run the ALTER DEFAULT queries with ...FOR USER postgres... in order for the readonly user to be able to read any tables created in db1 by the postgres user.

Well, we made it, after all that, I feel like we've really grown as people, and as friends. Hopefully this little trip into the abyss that is Postgres's permissions has been helpful :)


  1. There is so so so much more that could be talked about, but in the interest of simplicity I've assumed you would use the default public schema and that you don't care about things like search paths

  2. Unless you read this guide, then I've got you covered! 

  3. In actuality, I don't think they are entirely interchangeable, see section How to Define Privileges Upon Role Creation 

Constraints Breed Creativity

These are my slides/notes from a First Opinion company all hands presentation I gave in February 2016


FO-Constraints-Teaching-Moment.001.jpeg

What do you think of when you think of constraints?

FO-Constraints-Teaching-Moment.002.jpeg
Buy a tiny house they said...you'll save so much money they said

Maybe you think of this?

FO-Constraints-Teaching-Moment.003.jpeg

Or this?

FO-Constraints-Teaching-Moment.004.jpeg
look at all these floating things above my head, I'm creative

I think of creativity, because constraints breed creativity.

FO-Constraints-Teaching-Moment.005.jpeg

Look no further than music

FO-Constraints-Teaching-Moment.006.jpeg

12 notes, that's all you get! These 12 notes give us everything from Beethoven's 5th symphony to Hanson's MMMBop, and everything in between. They all use the same set of 12 notes1

FO-Constraints-Teaching-Moment.007.jpeg

Another example...

LEGOS!!!!! Simple lego bricks like this…

FO-Constraints-Teaching-Moment.008.jpeg
Like the giving tree, he gave all so I could finish my yellow Lego schoolbus

Give us things like this...

FO-Constraints-Teaching-Moment.009.jpeg

or this...

FO-Constraints-Teaching-Moment.013.jpeg
I can almost hear the Full House theme in my head

And, of course, because we’re in San Francisco, I couldn't resist this last example.

Any other examples you can think of where constraints breed creativity?

FO-Constraints-Teaching-Moment.014.jpeg

So what happens when you don't have constraints?

FO-Constraints-Teaching-Moment.015.jpeg

Star Wars, 1977, the original. $11 million dollars with a 94% rotten tomatoes score.

FO-Constraints-Teaching-Moment.016.jpeg
Still one of the greatest movie posters of all time

Star Wars Episode 1 - The Phantom Menace. The very definition of no constraints. George Lucas had complete control, from script and casting all the way down the line to post processing.

$115 million dollars, 56% on rotten tomatoes.

FO-Constraints-Teaching-Moment.017.jpeg

There are no constraints on the human mind, no walls around the human spirit, no barriers to our progress except those we ourselves erect.

-Ronald Reagan

So why do we set deadlines? And pare down feature sets? And don't just hire more developers and designers and product people? Because we are, to some extent, erecting constraints to increase our creativity and productivity as a company.

From The Inmates are Running the Asylum:

In the 1980s and 1990s, Royal Farros was the vice president of development for T/ Maker, a small but influential software company. He says, "A lot of us set deadlines that we knew were impossible, enough so to qualify for one of those Parkinson's Law corollaries. 'The time it will take to finish a programming project is twice as long as the time you've allotted for it.' I had a strong belief that if you set a deadline for, say, six months, it would take a year. So, if you had to have something in two years, set the deadline for one year. Bonehead sandbagging, but it always worked."

FO-Constraints-Teaching-Moment.018.jpeg
I always kept a spare bowl ready but my Goldfish never did this, not once

Below is an excerpt from when we launched First Opinion

We released our first real public version to the app store right before Thanksgiving, which in retrospect might not have been the best idea since most of us left on vacation immediately after. McKay wanted to be very hands on with the matching in the first release. ... So when a new user signed up, McKay would get a notification, he would look over their details and decide which doctor would be right for them.

And since I was going to be visiting my wife's family, I wanted to make sure those matching notifications were rock solid, because if McKay wasn't getting notified, the user wasn't getting matched with a doctor. So I rigged the server to send an email, a text message, and a push notification for each new user that signed up.

Over the next couple of days, First Opinion steadily climbed the app store rankings, moving into the top five apps in the medical category, and McKay's phone blew up with notifications, three at a time, to the point where he couldn't get any sleep because his phone was buzzing every few minutes ... On the flip side, each of our doctors was getting inundated with tons of new users every hour, all with a question or two to ask.

During this time, we were operating under the gun, as we were working to reconcile our growth with the amount of doctors we didn't have (we launched with only a few doctors but gained thousands of users in those first couple of days).

But as we scrambled to handle the load we figured out some incredible features of our Doctor application that helped our doctors manage the load, these are features we still use to this day, but they were created while we were working under intense pressure (which is a constraint) to handle our user load with the few doctors we had at the time.

FO-Constraints-Teaching-Moment.019.jpeg

He told me a story of how Larry Ellison actually got efficiencies from teams. If a team wasn't productive, he'd come every couple of weeks and say, "let me help you out." What did he do? He took away another person until the team started shipping…

I'm Closing with this quote because I think it perfectly encapsulates why we need constraints.

Supplemental Material

A few months after I gave this presentation, this tweet came through my stream, which I think is relevant...

Road Runner and Coyote

I also got pointed to these Road Runner and Coyote Rules, as recorded by Jason Kottke in 2012:

  1. The Road Runner cannot harm the Coyote except by going "meep, meep."
  2. No outside force can harm the Coyote -- only his own ineptitude or the failure of Acme products. Trains and trucks were the exception from time to time.
  3. The Coyote could stop anytime -- if he were not a fanatic.
  4. No dialogue ever, except "meep, meep" and yowling in pain.
  5. The Road Runner must stay on the road -- for no other reason than that he's a roadrunner.
  6. All action must be confined to the natural environment of the two characters -- the southwest American desert.
  7. All tools, weapons, or mechanical conveniences must be obtained from the Acme Corporation.
  8. Whenever possible, make gravity the Coyote's greatest enemy.
  9. The Coyote is always more humiliated than harmed by his failures.
  10. The audience's sympathy must remain with the Coyote.
  11. The Coyote is not allowed to catch or eat the Road Runner.

And while the rules might not have actually existed, the cartoons--of which there are 48 shorts, a half-hour special, and one full length movie--follow them pretty closely. 11 rules, ~450 minutes of entertainment.

Parchment

One of my coworkers sent me this article after I gave this presentation, one library decided to stop sending another library papyrus, so the other library invented parchment (emphasis mine):

One of the Ptolemies’ most drastic schemes to strike down the Library of Pergamum was the sudden cut of its trade of papyrus with the city of Pergamon. The Ptolemies hoped that if the main component of books was limited and hard to obtain, it would prevent the Library of Pergamum’s collection from growing. However, Pergamon came up with an alternative. Roman writer and scholar Marcus Terrentius Varro documented the event: “the rivalry about libraries between king Ptolemy and king Eumenes, Ptolemy stopped the export of papyrus … and so the Pergamenes invented parchment.

While it’s not possible for Pergamon to have invented parchment since scriptures on stretched leather have been found earlier in the east, the lack of papyrus may have pushed the king to expand the use and development of leather as a writing material, Coqueugniot says. The word for parchment in Latin, “pergamīnum” literally translates to “the sheets of Pergamum,” she says.

Blaise Pascal

This feels like something about constraints but I'm not sure what, I'm including it though because why not? And the quote has a fun attribution history:

I wrote you a long letter because I didn't have time to write you a short letter

My favorite alternative quote is the Woodrow Wilson one on giving speeches from 1918:

“That depends on the length of the speech,” answered the President. “If it is a ten-minute speech it takes me all of two weeks to prepare it; if it is a half-hour speech it takes me a week; if I can talk as long as I want to it requires no preparation at all. I am ready now.”

Is our Intelligence because of our Communication Constraints?

From the Import AI May 29, 2017 newsletter (emphasis mine):

Constraints as the key to intelligence: Machine learning whiz & long-distance runner Neil Lawrence has published a research paper, Living Together: Mind and Machine Intelligence, that explores the idea that intelligence is intimately related to the constraints imposed on our ability to communicate.

...the gist of Neil’s argument is that intelligence can be distilled as a single number, which he calls an Embodiment Factor. This expresses the relationship between how much raw compute an intelligence can make use of at once, and how much it can communicate information about that computation during the same time frame. Humans are defined by being able to throw a vast amount of compute at any given problem, but then we can only communicate at a couple of words a second at most.

...The way Neil Lawrence puts it is that a computer with a 10 Gigaflop processing capacity and a communication capacity of about 1 gigabit per second has an embodiment factor of 10 (computation / communication), versus a human brain which can handle about an exaflop of compute with a communication limit of about 100 bits per second - representing an astonishing embodiment factor of 10^16. It is this significant compression which leads to many of the useful properties in our own intelligence, he suggests.

Some Tweets and Quotes Collected over the Last Few Years

From @AdviceToWriters:

"The way to get over creative block is to simply place some constraints on yourself”

And a couple from @mmayer, one:

"As a society moves from scarcity to abundance, the cultural value that is most likely to be underdeveloped is restraint." - @mmay3r

and two:

"The more primitive your building blocks, the more complex the structures are that you can build." - @mmay3r

Paul Krugman (emphasis mine):

I'm really into breakfast. I start almost every day with fairly brutal exercise – i'm 66 and fighting it; today that meant an hour-long run in the park. Breakfast, usually starting with yoghurt and fruit, is the reward

So one of the best things about coming home is that some seasonal fruits -- things that aren't available all year round, at least in version you'd want to eat – have arrived. Mangoes! Fresh figs!

Are these fruits better than other fruits? Objectively, no. What makes them so great now is precisely the fact that you can't get them most of the year. And that, of course, tells you that standard consumer choice theory is all wrong

The textbooks (mine included) tell you that more choice is always better. But a lot of things gain value precisely because they aren't an option most of the time. I'd probably get tired of fresh figs and mangoes if I could get them all year round.

...if you imagine that being rich enough to have anything you want, any time you want it, would make you happy, you're almost surely wrong. Limits are part of what makes life worth living.

David Perrell:

Good design is not just easy to use but hard to misuse. The constraints are as important as the features.

And Adam Michela, answering this question from Marc Hemeon:

Do design systems prevent creativity because they lock you in to a vibe? or do they enable creativity because they help you not rethink UI and patterns and open your brain to more options.

Responded:

After inventing Facebook & Airbnb’s design systems, new hires would ask me:

“Won’t this limit my creativity as a visual artist? 👩‍🎨”

And each time I’d say: “Yes. That’s pretty much the point.“

And a quote from the book Hit Makers:

There is a kind of magic in the idea that humans can express quasi-infinite ideas and emotions from a code consisting of twenty-six funny-looking shapes.

There's a scene in Apollo 13 where the engineers at NASA pour out on the table all the parts the crew has available to them:

apollo13-dump.png

And one of the engineers says:

we've got two find a way to make this, fit into the hole for this, using nothing but that.

apollo13-fit-this-in-this.png

  1. If you want to you can constrain notes even more, and look at how much creativity you get, from Hit Makers:

    There is a popular online video called “4 Chords,” with more than thirty million views, in which the musical comedy group the Axis of Awesome cycles through dozens of songs built on the same four chords—I–V–vi–IV. (In the key of C-major, that progression is C–G–Am–F.) This chord progression is the backbone of dozens of classics, including oldie hits (the Beatles’ “Let It Be”), karaoke pop (Journey’s “Don’t Stop Believin’”), country sing-along (John Denver’s “Take Me Home, Country Roads”), arena rock (U2’ s “With or Without You”), animated musical (The Lion King’s “Can You Feel the Love Tonight”), acoustic pop (Jason Mraz’s “I’m Yours”), reggae (Bob Marley’s “No Woman, No Cry”), and modern dance pop (Lady Gaga’s “Paparazzi”).

    In 2012, Spanish researchers released a study that looked at 464,411 popular recordings around the world between 1955 and 2010 and found the difference between new hits and old hits wasn’t more complicated chord structures. Instead, it was new instrumentation bringing a fresh sound to “common harmonic progressions.” Several music critics use videos like “4 Chords” to argue that pop music is simply derivative. But this seems backward. First, if the purpose of music is to move people, and people are moved by that which is sneakily familiar, then creative people should aspire for a blend of originality and derivation. Second, it’s simply wrong to say that all I–V–vi–IV songs sound the same. “Don’t Stop Believin’” and “No Woman, No Cry” and “Paparazzi” don’t sound anything alike. These songwriters aren’t retracing each other’s steps. They’re more like clever cartographers, each given an enormous map, plotting new routes home.

I've Always Loved this Description of Hollywood Accounting

From The Last Don:

"I've never understood how I can have ten percent of the profit of a picture that grosses one hundred million dollars and costs only fifteen million to make, and then never see a penny. that's one mystery I'd like to solve before I die."

"It's absolutely legal," she said. "They are abiding by the contract, one you should not have signed in the first place. Look, take the one-hundred-million gross. The theaters, the exhibitors, take half, so now the studio only gets fifty million, which is called the rentals."

"OK. The studio takes out the fifteen million dollars the picture costs. Now there's thirty-five million left. But by the terms of your contract and most studio contracts, the studio takes thirty percent of the rentals for distribution costs on the film. That's another fifteen mil in their pockets. So you're down to twenty mil. Then they deduct the cost of making prints, the cost for advertising the picture, which could easily be another five. You're down to fifteen. Now here's the beauty. By contract, the studio gets twenty-five percent of the budget for studio overhead, telephone bills, electricity, use of sound stages etc. Now you're down to eleven million. But the Bankable Star gets at least five percent of the rentals, the director and producer another five percent. So that comes to another five million. You're down to six million. At last you'll get something. But not so fast. They then charge you all the costs of distribution, they charge fifty grand for delivering the prints to the English market, another fifty to France or Germany. And then finally they charge the interest on the fifteen million they borrowed to make the picture. And there they lose me. But that last six million disappears."

Remember you always want gross, not net.

It's Never Too Late to Start Something New

When I was a kid, Nintendo was the king of video game consoles, no one, not even Sega, could touch 'em. If you would've told me that within a decade Nintendo would be the third ranked console game maker and Sega wouldn't even be in the fight I would've called you a flat out liar, right there to your face.

It was inconceivable that Nintendo could fall so far, so fast. And now, while my daughter and I sometimes enjoy a mean game of Mario Kart Wii, chances are high my children will grow up never asking for Nintendo anything for Christmas or birthdays, something that a kid from my generation couldn't even imagine1.

I bring up this example because there seems to be a pervasive thought among people I know that it's too late to start something, every idea has already been done to death, every thought written down. So let's see if that really is the case, shall we?

No Way I could build a billion dollar company

You might be interested to know that the Fortune 500 turns over about 30 companies annually. Here's the top 10 on the list over three different periods.

Rank 1994 2004 2014
1 GM Walmart Walmart
2 Ford Exxon Exxon
3 Exxon GM Chevron
4 IBM Ford Berkshire Hathaway
5 GE GE Apple
6 Mobil Chevron Phillips 66
7 Altria ConocoPhillips GM
8 Chrysler Citigroup Ford
9 Texaco IBM GE
10 Dupont AIG Valero

While there are some companies with staying power, there are also new additions. In fact, this is entirely normal:

Because no company, no matter how successful, lasts forever, and because only a fraction of companies survive more than a few decades, turnover of varying degrees is entirely natural.

Because this ebb and flow of new companies onto, and off, the list is a natural occurrence, it means there is always time to start a new company.

It's impossible to make a living as an Author

This one is near and dear to my heart, not just because it's an inside joke in my family, but because my brother-in-law actually just published his first novel. Let's take a look at the best selling authors over three different periods2.

Rank 1994 2004 2012
1 The Chamber by John Grisham The Da Vinci Code by Dan Brown Fifty Shades of Grey by E.L. James
2 Debt of Honor by Tom Clancy The Five People You Meet in Heaven by Mitch Albom The Hunger Games by Suzanne Collins
3 The Celestine Prophecy by James Redfield The Last Juror by John Grisham Fifty Shades Darker by E.L. James
4 The Gift by Danielle Steel Glorious Appearing by Jerry B. Jenkins and Tim LaHaye Fifty Shades Freed by E.L. James
5 Insomnia by Stephen King Angels & Demons by Dan Brown Catching Fire by Suzanne Collins
6 Politically Correct Bedtime Stories by James Finn Garner State of Fear by Michael Crichton Mockingjay by Suzanne Collins
7 Wings by Danielle Steel London Bridges by James Patterson Diary of a Wimpy Kid: The Third Wheel by Jeff Kinney
8 Accident by Danielle Steel Trace by Patricia Cornwell Fifty Shades Trilogy Box Set by E.L. James
9 Disclosure by Michael Crichton The Rule of Four by Ian Caldwell and Dustin Thomason The Mark of Athena by Rick Riordan
10 Remember Me by Mary Higgins Clark The Da Vinci Code: Special Illustrated Collector's Edition by Dan Brown Gone Girl by Gillian Flynn

As you can see, there is a ton of turnover on those lists3, which means there's plenty of room for your great american novel. So you know, get to work!

I could never build a best selling iOS app

This one is for me, not only because it's my job, but I've even been working on my own app while learning Swift. And it's easy for me to think there is no possibility of my app breaking out, but4...

Rank 2009 paid 2009 free 2012 paid 2012 free
1 Crash Bandicoot Nitro Kart 3D Facebook Angry Birds Clash of Clans
2 Koi Pond Google Earth Doodle Jump Skype
3 Enigmo Pandora Radio Fruit Ninja eBay
4 Bejeweled 2 + Blitz Tap Tap Revenge Classic Angry Birds Seasons Google Earth
5 iBeer Shazam Cut the Rope Google Search
6 Moto Chaser PAC-MAN Lite TuneIn Radio Pro Twitter
7 PocketGuitar Backgrounds Monopoly Paper Toss
8 Flick Fishing Touch Hockey Angry Birds Rio Bump
9 Tetris Labyrinth Lite Edition FatBooth BBC News
10 Texas Hold'em Flashlight Flight Control Shazam

Angry birds pretty much dominated 2012, but if you go check the overall top app lists right now (don't worry, I'll wait) you'll see Angry Birds is no where to be found in the top 10 anymore.

So what's your point?

Take any other industry or segment and you will likely find similar churn5. This isn't to say your idea will rise to the top, or that it will be easy. On the contrary, it will take a tremendous amount of work, lot's of concentrated effort, and a little bit of luck (the actual luck and the self made kind). This was just to show you that nothing is set in stone, and things do change, in fact, they seem to be changing faster than ever. And as my wife so succinctly put it, "there's always room for the next big thing."

Supplemental Material

Some other things I've found since this was written.

Jerry Neumann

The 25 largest companies by market cap every 15 years from 1927-2017. The ones in yellow are those that were top 25 in 1927, or their offspring. Changing the guard is slow, then sudden.

25largest-1927-2017.jpeg

Michael Batnick:

In 1990, eight of the ten largest companies in the world were from Japan. Today, eight of the ten largest are from the United States.

1990-2018.png
source

Travis Fairchild:

here is a look at the number of Top 10 names that have historically drifted from the largest 10 over the next decade. The average in the U.S. has been around 6-7 on average.

top10fallout.jpg

Patrick McKenzie in 2019:

Question I get fairly frequently: is it still possible to make money in startups or as a creator on the Internet, or has the opportunity passed or been sewed up by AppAmaGooBookSoft or similar?

The answer to this: unambiguously, yes, opportunities are changing but better.

You should prefer being a founder in 2019 to being in the same position in 2009 or 1999. The larger market, much better ecosystem, existence of mobile, better OSS, etc swamp the impact of year to year availability of capital or increased competition or what have you.

Now given that you may not have a time machine available and aren’t choosing between those periods but are actually choosing “start now” or “start later” I think it’s a nuanced conversation but in general I’d be in the “bias towards earlier” camp.

The nuance is mostly about your relative rate of skill/network/etc growth and leverage available at whatever the thing you’re doing right now versus the likely higher-but-different skill growth and very variable leverage available from running something.


  1. I asked for a Nintendo every year for years, then a Super Nintendo. 

  2. Data was compiled with Wikipedia 

  3. And seriously, was I the only one that had no idea 50 Shades of Grey had more than one book? 

  4. This was the hardest data to find, and the problem with it is I think things like Facebook drop off the list because everyone downloads them once and has them on their phone forever more. 

  5. I also compiled a list of best selling albums, but I'm sick of making tables. 

Rinse...Repeat

You notice a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are.

You start small, a simplified interface that emphasizes function over form, no ads to annoy your users and a liberal use policy that makes it easy for your users to fit your product to their needs. People start talking about you, and more importantly, they start using your product. A small but growing group of users can't get enough and start telling their friends about how great your product is over all the other products in the market.

Every month more users keep coming, favoring the ease of use and no hassle simplicity of your product over your competition, not to mention the lack of ads your product currently has. You hire more people, you add more servers, and people just keep coming, they love your product.

After a while, you're huge, you're now the main product on the block and all your competition has faded away into obscurity and also-rans. Some Venture Capitalists take notice and approach you to see if you can turn your product into a real business.

You believe what they are selling you and raise a huge round of funding. You're different, they say, you're special. Sure, your product is free and commoditized, but you've built an audience. You're going to transform all those eyeballs into revenue.

In order to do that though you'll have to make a few changes. You remove a feature here, tweak a policy there. All necessary changes in order to lock down your product for advertisers. A small group of users start to complain, but you don't pay attention. Your changes are working, you're monetizing your audience. You knew some would leave, but everything is going according to plan.

Fast forward an unspecified time in the future. Some random person notices a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are...

The Most Important Thing About Your Job

I remember a conversation with my wife about a year after graduating, we were living in Washington, D.C. at the time and I was working as a Patent Examiner, with a plan to attend law school the following year, I'll be kind and just say that I didn't particular enjoy reading patents all day, and I wasn't particularly looking forward to law school. My wife was working at a law firm and was commenting about how unhappy all the lawyers at her firm were. She said something I've never forgotten, "I don't want you to come home everyday unhappy and talking about how much you hate your job."

We both quit our jobs a few months later and moved back west, I wanted to try my hand at starting a company because what I really wanted was the flexibility and joy that comes from building something from nothing. And what my wife really wanted was for me to come home from work happy each and every day1.

I bring this up because I've been recently working through some old hard drives that are just full of all kinds of things from the last twelve or so years of my life. One of those old documents contained an employment survey I had filled out in the distant past, the Career Consultation section was interesting.

personal-profile-career-consultation.png
I guess I'm not motivated by money

While I'm embarrassed about a few of those answers, I think it's interesting what I focused on when asked what is most important to me, where I wanted to be in a few years, and what my ideal job was. The only thing that mattered to me was to do something I enjoyed and was exciting to me.

Even back then, I wasn't motivated by money or prestige, I was motivated by the desire to enjoy what I do, because when you spend so much time at one place, and there are going to be so many bad things about any job, you need to really enjoy it when it's good.

Hopefully, everybody will strive for something better than just not hating your job, a good test to see where your true feelings lie about your current employment comes from Senator Schumar:

"I have a little test, when you wake up monday morning, do you feel in the pit of your stomach that you want to go to work. And if you can say yes to that, you're in great shape."

I hope all of you are in great shape :)


  1. Man I love my wife 

The Absolute Worst Business Model There Is

I think making something inconvenient to use unless they pay you is the absolute worse way to make money. Want to watch that movie on any device? Tough luck! Want to read that article? Well, you can't unless your a subscriber. Oh, you didn't think we would let you listen to that song anywhere you wanted, did you?

I would like to say the idea of making money through inconvenience is the exclusive domain of large dying giants like the RIAA, MPAA, and publishing conglomerates, but it's not, lots of startups fall into this same trap also.

Heck, we even tried inconvenience as a business model at First Opinion. We want First Opinion to be the first place people turn to when they have a medical question, and yet, what did we do? We limited your interaction with our doctors to once a month unless you gave us money. A move that all but assured we wouldn't be the first place you turned to when you had a medical question.

While we gathered extensive stats that told us this model just wasn't working for us, it really hit home as a problem when my wife, my own beautiful wonderful wife, said she had two more days until her free question renewed when I asked her why she wasn't asking her First Opinion doctor a medical question she had.

Our mistake was we were making people stop and think about what they should do, as Influence: The Psychology of Persuasion notes, this is a huge no no:

The renowned British philosopher Alfred North Whitehead recognized this inescapable quality of modern life when he asserted that "civilization advances by extending the number of operations we can perform without thinking about them."

Evan Williams, of Blogger, Twitter, and Medium fame, expanded on this in an XOXO talk he gave in 2013:

"The internet makes human desires more easily attainable. In other words, it offers convenience [and] Convenience on the internet is basically achieved by two things: speed, and cognitive ease [...] If you study what the really big things on the internet are, you realize they are masters at making things fast and not making people think.

"Here's the formula if you want to build a billion-dollar internet company [...] Take a human desire, preferably one that has been around for a really long time [...] Identify that desire and use modern technology to take out steps."

In other words, people would much rather pay for convenience over inconvenience. So, if you're trying to decide what business model to pursue, I would encourage you to choose making your product more convenient, even if it results in less money at the start, because chances are, doubling down on convenience will make you larger in the long run than a model of inconvenience ever would.

This is exactly what we've observed at First Opinion, we dropped our inconvenience model and made access to a First Opinion doctor free for everyone, anytime, and instead decided to focus on charging for additional conveniences like speed of our doctor's responses and the ability to send photos to our doctors.

And if you're a large dying giant that has decided to go the inconvenience path, remember this observation by Clayton Christensen and hopefully take heed:

In nearly every instance of disruption we have studied, the survival instincts of the disruptees—the prior industry leaders who are being disrupted—set in motion defensive actions intended to slow the pace of disruption. In the end, however, the advantages that disruptive competitors bring to customers in terms of quality, cost, convenience, and accessibility become so apparent that the regulations are removed and the disruption proceeds apace.

The Miracle of Birth

My sister has had a few home births. I've got a niece that is basically in and out of the hospital in less than a day with new baby in hand. Heck, there was even a couple that used First Opinion to deliver their baby in the backseat of their car.

Unfortunately, easy baby arrivals doesn't seem to be how my wife, Dee, rolls. Our daughter was minutes from being taken by C-Section when she finally arrived, and so I guess I shouldn't have been so surprised when a routine doctor visit a few weeks before the due date went too long1.

Wednesday, July 16, 2014, 5:04pm to 5:14pm

Me: Just curious where you're at?

Dee: Still at the doctor. I think everything is fine but it's a long story. Hopefully coming home soon.

Me: Eeeek

Dee: Don't freak out. I think it's ok

Me: Too late, I'm freaked, you've been at the doctor for over 2 hours

Dee: It's really not as bad as it seems. Just too long to tell you over text. It does look like the baby is breech though. Or at least turned funny

Dee: Doesn't mean he's not healthy or that it won't be ok though.

Me: :( let me know when you're headed home

Dee: Ok

Now I want you to close your eyes and imagine the doctor just told you the baby is breech and the umbilical chord is hanging down so if you went into labor outside a hospital you would most likely lose the baby. Now, imagine the doctor wants you to go the hospital to have them attempt something called a version, which you've never heard of, where they attempt to re-orient the baby into the correct position while still in the womb. Finally, imagine you had planned to have the baby in Redwood City, you knew that, your husband knew that, and your doctor knew that. Okay, so you need to go to the hospital, what hospital do you go to?

If you're my wife and me? You pile into the car, with makeshift overnight bags, and drive to Redwood City, walk in, go up to Labor and Delivery, where there is only a door that says Stop! No Entry and then back down to admitting, where the helpful clerk tells you you need to go back up to the floor you were just on and through the door that says Stop! No Entry and then down the hall and to the left where the nondescript unmarked door that has a silver buzzer hides Labor and Delivery.

The nurses were not expecting us, which is a seriously bad sign in a situation like this. After some back and forth, we were informed that versions are only performed at the San Francisco hospital, and after a quick phone call it was confirmed that San Francisco was wondering where the heck we were. We would've had a good laugh about the misunderstanding if any of us had been in a laughing mood, maybe someday.

My sister met us at Redwood City and took our daughter, Kenzie, home with her while Dee and I began the long rush hour drive back up to San Francisco. We finally arrived at the correct hospital a little before 9pm, where we were educated about what was about to happen. The doctors would attempt to turn our baby, if that failed, they would prep Dee for a C-Section and put her under anesthesia, and then try again more forcefully to turn him.

If either of those were successful, no C-Section for us. We were also informed versions have only about a 50 percent success rate. Also, they can't use the medication they normally use for the procedure because it elevates heart rate and Dee's heart rate was already a little too high, so our success rate was even lower.

Wednesday, July 16, 2014, 10:23pm

Me: Version attempt 1 will take place soon

My Sister: Good luck and fingers crossed

At about 10:30pm the doctors began the first version attempt. My sister tells this great story about a massage she got in India once2, where she sat in the middle of a room, cross-legged and naked, while someone poured oil over her head and then basically punched her back. She called it the worst massage any person could ever get ever, a version is kind of like that. I pressed my forehead to Dee's forehead while she grunted through the pain of two doctors forcibly pushing on her stomach trying to knock our baby back into the correct position.

Wednesday, July 16, 2014, 10:55pm

Me: Wow, they were able to turn the baby in one of the most uncomfortable massages I've ever witnessed.

My Sister: I can only imagine. Will they monitor through the night?

Me: Now they are going to watch for a few hours to see if she will continue to stay in labor or will return to normal

My Parents: Wonderful! But sorry it was hard.

Me: The doc says he might be inclined to induce since she is one day short of 38 weeks

Me: He's not sure he wants to risk the baby turning again

My Sister: Makes sense.

My Parents: How is dee doing? Give her our love.

Me: She says she doesn't know how she's doing, it was just supposed to be a normal checkup

The next 13 hours are standard labor and delivery fair, if this was a movie, here is where the montage would be. Dee's water broke around 5:49am, they put an epidural in around 7am and prepped the delivery table. They moved Dee into position at about 10:30am, and Hayden was born about 11:44am, an 8 lb 10 oz exciting new addition to our family. Between events I was able to grab a little bit of sleep, and Dee was not.

Thursday, July 17, 2014, 11:44am

Me: Hayden j Marcyes, no idea weight or time but he's here

My Parents: We are so happy. Now you can both get some rest.

My Other Sister: So happy all is well!!!

Me: 8 lb 10 oz

My Other Sister: That's pretty big. Holy cow.

My Parents: Good thing he didn't wait two more weeks

My Sister: Big boy...he looks great. I am tearing up.

kenzie-hayden.jpg
My wife grows cute kids

That makes two, count them, two narrow C-Section dodges. And two freaking awesome kids.


  1. I noticed around 5pm that my wife hadn't returned from her 2:30pm appointment. 

  2. which I am about to completely murder with my abbreviated retelling, when my sister tells this story my stomach hurts from all the laughter. 

The Only Trend That Matters For Your Startup

This post was originally published in 2012 on the Startup Grind blog, I'm republishing it here for archival purposes, also, my wife gave birth to our second child this week so there was no way I was going to have the time to write a new blog post :)

04-oink.jpg
Oink Oink

I was quite surprised to read all the negative comments about Milk shutting down Oink1. The gist of the armchair quarterbacking is Oink had lots of users and so Milk should be obligated to support it into perpetuity because, well, they never gave a good reason.

Social apps are hard! When you launch a social app like Oink you need to have a great growth story almost from day one. Sure, Oink grew quickly on the strength of Kevin Rose and Daniel Burka's much deserved reputations, but after the dust settled, I'm guessing their true growth rate settled into a pretty consistent linear rate and they realized, rightly so, that this will never be a phenomenal success, and so they decided to move on, I applaud that.

You need to watch your growth trend line carefully, Ben Silberman, a few weeks years back talked about how Pinterest grew 50% consistently month over month. This is a great growth rate2. Even though it still takes a few years when you start with a small user base (in Pinterest's case, it was about 200 users) you are adding radically more users each month than the one before it, this is what you want.

I'm guessing Oink was the exact opposite, after their initial pop, I bet growth settled into something more regular and less impressive. They managed to quickly get 150,000 or so users, and then their true growth rate settled at maybe 5-10,000 new users a month, this is the nightmare scenario.

Most people--that have never tried to build a venture backed social app--probably think growing by 5-10,000 users a month is awesome, and if you were charging money for something, it probably would be, but for a free social product where users are considered revenue, a linear growth rate is the first knock at your door by the Grim Reaper. Starting from a user base of 150,000, and growing at 10,000 user a month, it would take about 7 years to get to a million users, still think that's good growth?

Now let's look at exponential growth. Starting with 200 users, and growing 50% every month, it will take just shy of 2 years to hit a million users, and if your growth rate stays around 50%, that's when your growth would really start to take off, adding hundreds of thousands, and then millions of users each and every month. See the difference?

Remember, Milk was a free product, its users were its revenue, and its revenues weren't growing fast enough to make the company viable. I think it's great the whole Milk team recognized they had a bad growth trend line and chose to move on3, those kind of actions should be commended, not ridiculed.


  1. Oh no, this post isn't dated at all /sarcasm 

  2. I used this same example in my 2013 Startup Grind presentation 

  3. Remember, this was originally written before Milk was acquired by Google and Rose was a VC. 

This One Lesson Made College Worth It

I didn't have a lot of programming experience before starting college. I'd loved computers since my parents had brought home our Mac plus around 1985, and I spent hours in my youth configuring, tinkering, and playing1 with the steady string of computers my family owned2, but I never actually programmed anything using them.

That all changed during my first semester at BYU, when I took the introduction to programming course. It was hard, but fun, and I managed to power through it with few problems, and I even did pretty well overall, so I figured this programming thing was going to be easy and with a few hours of work here and there I could conquer it.

Then came the second programming class. The introductory class used an integrated Windows IDE that streamlined the editing, compiling, and running of my code into one easy to use package. That second class, however, switched to Linux and required using the command line for compiling and running my code.

It was a night and day change for me. They moved really fast through the material3 because we all had previous programming experience now. I didn't know Linux, I didn't know what text editor to use to write my code. I didn't know how to compile my code using the command line, let alone run it. Every. Freaking. Thing. Was. Different. And To top it all off, my Grandpa died early in the semester, and I missed about a week of classes.

I got behind, and I mean really behind, and it became obvious when I sat down in the Linux computer lab, the day before my first project was due, and struggled to even open a file. It was my worst nightmare. I remember feeling sick when I realized I didn't understand half the project description and that that there was no way I was going to finish the project before it was due. And then I noticed the second and third projects built onto the first, so I was just going to get further and further behind. There might have even been some tears.

The next week I spent nearly every waking hour in the computer lab making nearly zero progress. And it became apparent I wasn't going to be able to figure things out on my own like I always had before. I was too crunched for time, too far out of my comfort zone, and under too much pressure. So I did something I had never done before, I went to the TA help lab and asked for help.

I'm not going to lie, it hurt my pride quite a bit to walk into that help lab and start asking the kind of questions I was asking4, especially already being a week late on the first project and about to be late on the second project. But the TAs helped me choose a text editor, design my project, and write and compile my code.

It took me about five weeks, and lots and lots of hours in the computer lab, to completely catch up, and I spent a lot of time getting help from the TAs. But I've carried that experience with me to this day. It's the single best thing I learned while getting my Computer Science degree, and it's definitely served me better than anything else I learned, I learned how to ask for help.

This tweet from Maddie Stone (Security Researcher at Google) made me think of my college experience:

2nd semester of CS, I asked "What is a server?" in my intermediate program class.

Everyone laughed, the professor looked at me incredulously & I continued with "What do you mean a computer doesn't have a screen?"

And here we are today. Ask the questions until you understand.


  1. Dark Castle, King's Quest, Space Quest, and Police Quest, I love you all! 

  2. We started with the Mac Plus, then we switched to a Pentium 386DX running Windows 3.1, only to switch back to a Macintosh Quadra 660AV right before the Powermac's came out :( 

  3. Things like module importing, namespaces, and Polymorphism were completely new to me, the first course basically never required more than one file or more than one class, and then I was all of a sudden required to have multiple files, with multiple classes. 

  4. As a former TA myself, questions like: How can I open a file? and How do I compile my code? are not the kind of questions you want to hear from a student that is already a week late on the first project.