These are my brushed up notes for a presentation I gave during one of First Opinion's all hands meetings in October 2015.
Types of Mistakes
I want to start with a discussion of the three types of mistakes, as relayed to me by a good friend of mine:
Honest mistakes - Everyone falls victim to an honest mistake sometime in their life (I fall victim to them more than most). You accidentally push incomplete code to production or you leave an ad campaign running just a little too long. We're human, these mistakes happen, so we fix them and move on.
Incompetent mistakes - The people that are committing these mistakes are either out of their league, or incompetent. Either way, it's time to cut them loose or give them a different job.
Process mistakes - Most mistakes are because of a lack of process, so I want to spend the majority of my time talking about this type of mistake and how we seek to minimize them (nobody's perfect) at First Opinion.
Minimizing Process Mistakes
Let's start with Engineering, since I happen to know a thing or two about how the engineering team works.
The first line of defense for engineering is our automated tests. Each major piece of our codebase has a decent amount of tests backing it up, as a completely unscientific informal look at the amount of code we have written just to test the codebase, here are some conservative stats on our three biggest software areas1:
- Server - 500+ tests comprising around 15,000 lines of code.
- iOS - 60+ tests comprising over 10,000 lines of code.
- Web-client - Many tests comprising around 5,000 lines of code.
- Other - We have multiple open source projects that also have decent testing suites.
We are constantly adding new tests and making sure the existing tests are still relevant so our Engineers can have that sweet sweet piece of mind that comes from knowing any changes they make to the codebase don't ripple outward like silent chaos ninjas to cause unforeseen bugs elsewhere in our system because without automated tests...
Making even small changes will become increasingly difficult. Eric Evans in Domain-Driven Design: "When complexity gets out of hand, developers can no longer understand the software well enough to change or extend it easily and safely." Facebook [needs] a huge staff to keep up their momentum maintaining a big ball of mud. ... Releases will break things, because you don't understand the relationships well enough to pretend the impact of your changes. ... Next time management or clients try to convince you to move faster and throw quality under the bus, you can say sure, that will work, as long as you can hire 429 engineers to work on our iOS app. via
In fact, our iOS engineers just spent the better part of the last two weeks knocking down the last untested part of the iOS application in an effort to release the most stable app we've ever released2.
Jarid also recently spent a solid week figuring out how to automate the testing of a new feature we're working on3. I guarantee our competitors didn't devote that much time to ensuring they could automatically test that feature, that's what makes us different.
But wait, there's more. The Engineers also do code reviews, these are when one engineer has another engineer look over their code and make suggestions on how it can be improved. These are incredibly valuable in helping all the engineers understand the codebases and helps the codebase stay high quality and well documented since your best critic isn't yourself, it's your colleague who has to maintain your bug infested code.
We try and automate all the things. When we deploy, we deploy with one simple command. When we add servers to our system, that's also one command. Automation means we do things the same way each and every time, thus minimizing mistakes.
When we do find mistakes, we fix the automation scripts, for a change once, fix everywhere workflow.
We have a full time user researcher that is constantly going out and talking to people about our product and planned upcoming features. Think of that for a minute, we have someone whose whole job is to go out and speak to real people about our product, do you know any other company that does that?
Before any major feature gets anywhere near engineering, it's gone through multiple rounds of user testing using paper wireframes, Then onto small little interactive prototypes on the phone, and only after all that is the feature ready to be passed to engineering to be built.
On the doctor side
Each of our main doctors has a staff of support doctors that help them out, our matched doctors review the conversations and monitor the quality of the interaction each one of our users has on our service.
We also have a full time QA staff that works continuously to make sure each and every interaction with a Doctor on our app is a high quality one.
Besides that, we ask the user's themselves to rate their interaction with our doctors and take that feedback incredibly seriously.
But things always go wrong
It's true, they do, and we're no exception. So when a user does have a bad interaction, we kill them with kindness, and we work with them personally to make sure their issue is resolved to their satisfaction.
"Reverse engineer your successes and turn best practices into best processes"
Whenever we have problems, we go through what caused the problem and talk about how we fixed it and what we are going to do different to make sure we never see the problem again, this is the best way to make sure we are always solving new problems and moving forward, instead of going insane.
In Engineering, we also change the code, add tests, or automate the problem away, the goal of these post-mortems and the implementation of the solutions is to minimize the same mistake happening a second time.
We also do one more thing...
We pass it on down
Anyone familiar with the Banana story?
Start with a cage containing five monkeys. Inside the cage, hang a banana on a string and place a set of stairs under it. Before long, a monkey will go to the stairs and start to climb towards the banana. As soon as he touches the stairs, spray all of the other monkeys with cold water. After a while, another monkey makes an attempt with the same result - all the other monkeys are sprayed with cold water. Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.
Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and wants to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted.
Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm! Likewise, replace a third original monkey with a new one, then a fourth, then the fifth.
Every time the newest monkey takes to the stairs, he is attacked. Most of the monkeys that are beating him have no idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey.
After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water. Nevertheless, no monkey ever again approaches the stairs to try for the banana.
Because as far as they know that's the way it's always been done around here.
While this story usually has a negative connotation, it doesn't have to be that way, passing down good company culture is a great thing and helps makes us stronger.
We learn from each other, hopefully we retain mostly the good stuff and fix the bad stuff.
This fanatical obsession with scalable quality assurance has been going on from the very first line of code and the very first user/doctor interaction, to where we sit now at hundreds of interactions a day, to the future at thousands of interactions, and tens of thousands of interactions, and millions of interactions4. We focus so much on our internal quality because we know our internal quality eventually becomes our external quality.
This is what makes us different, this is what makes us special, and this is what will help us change healthcare.
This was incredibly unscientific, for example, in the iOS repo, I went into the testing directory and ran:
find . -name "*.m" | xargs wc -lto get the line count. ↩
Which we did release...in 2015, a release about 100x more stable is still available in the app store. ↩
Sadly, I removed the specifics of the feature and now I'm not 100% sure what feature it was. ↩
Fun fact, we've hit most of these interaction goals over the last few years. ↩
On Tuesday, November 8, 2016, we voted, and Donald Trump will be the next President of the United States.
While watching the coverage of the election over the last year, I couldn't understand how any rational person could ever vote for someone like Trump, but even the most rational of people can work against their best interests when they feel unfairly treated, as demonstrated in an economic experiment called the ultimatum game1:
The first player (the proposer) receives a sum of money and proposes how to divide the sum between the proposer and the other player. The second player (the responder) chooses to either accept or reject this proposal. If the responder accepts, the money is split according to the proposal. If the responder rejects, neither player receives any money.
The responder typically vetos the deal when low offers are made. People consistently prefer getting nothing to receiving a small share of the pie. Rejecting the offer is in effect paying to punish the proposer.
The reason why this experiment is interesting is because in a completely rational world, the responder would accept any amount of money the proposer offers, because anything is better than nothing, right? Not so fast, we let emotion and passion guide us all the time, especially when we think we're getting a raw deal or feeling betrayed, as Ben Thompson wrote in his daily update after the election2:
People feel the system is unfair, that they don’t have a stake in it, and they are exceptionally mad. This is a sentiment that has been rising for decades thanks to globalization and the increase in manufacturing productivity that devastated so much of the Midwest in particular, but it was dramatically accelerated by the aftermath of the Great Recession [of 2008] when, in the estimation of people where I grew up, Wall Street got off scot-free while everyone else had to bear the pain. I cannot overstate how pervasive and deeply-felt this sentiment is: so many people I know — regardless of income level — deeply believe the system is rigged, so who cares if Trump destroys the whole thing?
David Wong says the same thing (seriously, if you only read one external link from this post, read this one) with a little more color:
They're getting the s**t kicked out of them. I know, I was there. Step outside of the city, and the suicide rate among young people f**king doubles. The recession pounded rural communities, but all the recovery went to the cities. The rate of new businesses opening in rural areas has utterly collapsed.
See, rural jobs used to be based around one big local business -- a factory, a coal mine, etc. When it dies, the town dies. Where I grew up, it was an oil refinery closing that did us in. I was raised in the hollowed-out shell of what the town had once been. The roof of our high school leaked when it rained. Cities can make up for the loss of manufacturing jobs with service jobs -- small towns cannot. That model doesn't work below a certain population density.
If you don't live in one of these small towns, you can't understand the hopelessness. The vast majority of possible careers involve moving to the city, and around every city is now a hundred-foot wall called "Cost of Living."
This sense of loss and unfairness has been rising for decades, but it feels like it has only been in the last eight years or so that this unease has reached the tipping point and started to manifest itself politically, Game Change touches on this unrest boiling to the surface during the 2008 elections:
As the election barreled toward its conclusion, something dark and frightening was unleashed, freed in part by the words of the McCains and Palin. At rallies across the country, there were jagged outbursts of rage and accusations of sedition hurled at Obama...
Colin Powell had been friends with McCain for twenty-five years. The senator had been actively seeking his endorsement ... Powell warned McCain that his greatest reservation was the intolerant tone that seemed to be overtaking the Republican Party. McCain’s selection of Palin bothered Powell because he saw her as polarizing ... And then there were the hate-soaked rallies...
The general’s repudiation was a stinging blow for McCain ... Powell represented the same brand of Republicanism as McCain’s. Tough on defense. Fiscally prudent. Pragmatic and nondoctrinaire. McCain had to wonder what had become of him if his current incarnation was repelling someone like Powell. He was startled by the crazies at his rallies. Who were they? Why were they there? And what did they see in him?
And, indeed, the first Tea Party candidates were elected in 2010, marking the beginning of a new era of political unrest.
Just before the election we packed up the kids and headed south from San Francisco to Disneyland. On our way we passed many a barren field with a sign reading, "congress created dust bowl"
And oftentimes, right next to those dustbowl signs, was a "Farmers for Trump" sign:
Signs like this are usually preaching to the choir though, because as it turns out, America votes by population density, more or less
As cities continue to grow in red states, those cities will become more blue, and ultimately, those states will become more purple, and then blue...
If you follow the red state trend lines, you can clearly see that any dense, fast-growing cities that might emerge in red states will be very likely to vote blue. The few that do already exist already vote blue. How would these new cities be different and [what would] cause them to vote red?
Let's try and answer that question: what would cause cities and/or states to vote red? There really are only two answers:
- More rural voters showing up to vote
- less urban voters showing up to vote
Because if urban voters actually show up and vote, they would overwhelm any national election each and every time.
So how would you cause more rural voters to vote and less urban voters to vote? On the rural front, the answer seems to be a growing sense of unfairness, tinged with a fair bit of legitimate anger, and topped with a helping of more rural voters actually bothering to vote, in larger and larger numbers, in each election:
In 2008 and 2012, rural voters accounted for only about 19 percent of the vote in the state, but according to preliminary exit poll results, they account for 27 percent of the state’s vote. And Trump does well with those rural voters, winning them by about 15 points.
Trump leads Democratic rival Hillary Clinton by the commanding margin of 55 percent to 18 percent among farmers with operations of at least 200 acres.
Why would a lifelong urbanite ... enjoy such robust support in the heartland? The answer may lie in The Donald's unmatched ability to deliver an angry, frothing rant. The Agri-Pulse poll finds that 86 percent of respondents reported being somewhat or very dissatisfied with the way things are going in the United States...
As Siena Chrisman recently put it on Civil Eats, "Rural America is mad." She notes that Trump polls strongest in places where "white identity mixes with long-simmering economic dysfunctions," quoting a New York Times analysis. And few places combine embattled white identity and economic stress quite like farm country.
And how would you get urban voters to not show up? Choose some candidates no one is particularly excited about
Look at the effect of urban voters not, you know, actually voting:
In Michigan, where Clinton lost by around 13,000 votes, some analysts estimate that 90,000 Democrats left the top line blank.
Now take the results in Michigan and multiply them across the entire country:
Notice anything? The incredibly shrinking number of blue counties by chance? Particularly among the rust belt cities, which have been hit really really hard over the last 8 years and maybe aren't enamored with the current status quo? So now we know, that's how you get previously blue voting counties to become red.
Now, let's look at some trends, why does rural vote red and cities vote blue? Well, for one, we seem to actually be physically separating ourselves:
heavily urban counties ... were 32 percentage points more Democratic in their voting than the average. The heavily rural counties were about 11 percentage points more Republican...
If you plot every county's urban-versus-rural divide by the per-election average change in the vote, the pattern is clear: more urban areas vote have been voting more Democratic...
What's particularly interesting is to see how each county has changed over time. Notice that urban areas, counties containing cities, have grown more Democratic; the counties surrounding them (usually suburbs) less so....
the political shifts that have emerged as people of like-minded politics have moved into similar and dense areas ... over time, the tension between rural and urban areas in every state could become significant factors in politics, if this polarization continues.
At the same time we're also separating ourselves virtually as well (and the internet makes everything happen faster). Something Awful noticed this clear back in 2004, while talking about Furries and the empowering effect of the internet:
In the endless expanse of communications the Internet is, probably the greatest and most terrible gift it offers to furries ... is the ability to shut themselves off from the mainstream. They huddle in cloisters that are virtually unassailable by the outside world and whisper encouraging things to one another that would be nearly impossible to say in real life. Free from the pressures of society to conform to a boring standard they go in the exact opposite direction, externalizing things that are roughly as far from "normal" as can be expected. Then, within their protected virtual enclaves, they declare these things to be the norm. By declaring their perversions, mores, and general imbecility to be their own status quo they have simultaneously validated their own existence and demonstrated the inferiority of outsiders.
This makes it hard to see the other side's viewpoint:
It seems that people just don’t want to compromise on anything anymore, and it seems to be getting worse and worse with each and every year, people are clustering to their respective side and refusing to venture towards the middle for any reason...
When you can surround yourself with a community that only shares your beliefs, you have a recipe for killing any and all compromise, why should I ever try and see your point of view when all my peeps over here are yelling and screaming that you suck?
This is entirely in line with Ben Thompson's aggregation theory:
in an aggregated world it is voters aka users who decide which issues get traction and which don’t. And, by extension, the most successful politicians in an aggregated world are not those who serve the party but rather those who tell voters what they most want to hear.
Which also follows Ezra Klein's observations:
It’s tempting to imagine that rising political polarization is just a temporary blip and America will soon return to a calmer, friendlier political system. Don’t bet on it. Political polarization maps onto more than just politics. It’s changing where people live, what they watch, and who they see — and, in all cases, it’s changing those things in ways that lead to more political polarization, particularly among the people who are already most politically polarized…
It’s easy to see how this could work to strengthen polarization over time ... people become more extreme when they’re around others who share their beliefs. If liberals and conservatives end up moving to different places and surrounding themselves with others like them they’re likely to pull yet further apart. And even for those who can’t move, the internet makes it easy to settle in a virtual neighborhood with people who agree with you. Polarization is going to get a lot worse before it starts getting better.
And as people have separated into new segregated physical and virtual communities, they don't seem to be listening to traditional gatekeepers anymore:
"I've been endorsed by over 70 newspapers,” the Ohio governor [John kasich] said. “Wish it mattered."
Hillary Clinton also fell victim to how little traditional gatekeepers mattered:
Hillary Clinton had just about every single endorsement from newspapers and magazines across the country – as well as a majority of mainstream TV personalities – on her side, even if reluctantly. And it meant nothing at all.
And not only are people no longer listening to traditional gatekeepers, they are actively preventing established power structures from taking control from them:
Republican voters ultimately did go for Trump, however. In fact, they intervened to wrest control of the nomination back from the delegates. Until just a few weeks ago, Trump’s position had stagnated in national polls and in state-by-state results; before New York on April 19, he’d yet to win a majority in any state, lending credence to the idea that he might have a “ceiling” (albeit a higher ceiling than some originally expected) on his support. From New York onward, however, Trump has won a majority in every state ... with a sudden lunge upward after Wisconsin, rather than a smooth upward projection.
What happened after Wisconsin? My theory ... is that Republican voters were swayed by Trump’s arguments that the candidate with the most votes and delegates should be the nominee ... Some voters might have preferred Cruz or John Kasich to Trump in the abstract, but not at the expense of a contested convention in which the plurality winner would be denied the nomination and replaced with another flawed candidate.
A few weeks back I had the privilege of talking to a well known CEO, as the topic turned to politics, he talked about how Trump would never win, but when I pointed out that the people who voted for him don't actually go away when the election is over, he responded with, "yeah, what are they going to do about it?"3
A few days after the election, while talking to another CEO about the election, he asked, "what are we going to do about it?" Hopefully, this election marks a change of attitude for a lot of people (and a smidgen of empathy would be nice also), and the beginning of something great, because change rarely happens during times of political stability, but can come fast during times of instability, my hope is that any change is is ultimately positive, not just for a privileged few Americans, but for every american, in every state.
In the end, for all Trump's talk about how the election is rigged, I think we've just seen the best possible proof it is not. That ultimately, people will be heard, and how important each person's vote really is.
Paywalled, but worth the price of admission! ↩
When I started writing this post I thought Trump was going to lose, and I was going to use this story as a means of talking about how when people are really angry, what they do is start revolutions: like The American Revolution, and the French Revolution, and the American Civil War. ↩
So we needed to add a Postgres user that could only read tables from our database--not only the currently existing tables, but any tables added in the future--and I found Postgres's permissions system surprisingly complex and unfriendly.
Sure, everything you need to know is buried throughout multiple manual pages and tons of Stack Overflow answers, among other places, but I didn't feel there was anything out there that brought together all the disparate sources into one easy to follow beginner's guide to understanding Postgres permissions, so here we are, the easy guide to creating a permanent read-only users in PostgreSQL1.
Before we dive in, I'm using version 9.3 of Postgres, on Ubuntu 14.04:
$ psql --version psql (PostgreSQL) 9.3.11
So all this information will apply to that version of Postgres, and all the commands will work on that version of Ubuntu. I'm not sure what changes, if any, would need to be made for earlier, later, or different versions of Postgres and Linux.
It really helps to understand some of the different data structures that come into play when messing with Postgres's permissions, by default, every database you create has a default schema named public, the schema you use is incredibly important and could be a great source of frustration and annoyance2 as you mess with the users and try and set their permissions.
But it won't be frustrating for us because we're going to learn by example, on a fresh install of Postgres, let's create our two users, our owner foo and our read only user readonly:
$ sudo -u postgres psql -c "CREATE USER foo WITH ENCRYPTED PASSWORD '...'" -d template1 $ sudo -u postgres psql -c "CREATE USER readonly WITH ENCRYPTED PASSWORD '...'" -d template1
and let's have our new foo user own our two databases:
$ sudo -u postgres createdb -E UTF8 --locale=en_US.UTF-8 -O foo db1 $ sudo -u postgres createdb -E UTF8 --locale=en_US.UTF-8 -O foo db2
Now, let's create a table in db1:
$ psql -U foo db1 db1=> CREATE TABLE bar1 (_id INT); CREATE TABLE Time: 8.384 ms db1=> \q
and a different table in db2:
$ psql -U foo db2 db2=> CREATE TABLE bar2 (_id INT); CREATE TABLE Time: 9.042 ms db1=> \q
So now database db1 should have a bar1 table, and database db2 should have a bar2 table. Now we're ready to understand the public schema of the database.
What we are going to do is connect to db1 and list its tables, then connect to db2 and list its tables:
vagrant@vagrant:~$ psql -U foo db1 db1=> \d List of relations -[ RECORD 1 ]-- Schema | public Name | bar1 Type | table Owner | foo db1=> \c db2 You are now connected to database "db2" as user "foo". db2=> \d List of relations -[ RECORD 1 ]-- Schema | public Name | bar2 Type | table Owner | foo
Notice that each table belongs to schema public, but each public schema has a different table. So now we understand that when we mess with the public schema, we need to be cognizant of what database we are connected to because any changes we make to the public schema will only affect the public schema of the connected database.
The second thing we should undertand is Postgres seems to use role, group, and user interchangeably:
The concept of roles subsumes the concepts of "users" and "groups". In PostgreSQL versions before 8.1, users and groups were distinct kinds of entities, but now there are only roles. Any role can act as a user, a group, or both.
We're only ever going to use user here, but when looking at different sources of information, you might find people using role or, less likely, group3.
The readonly user
We want the readonly user to be able to read all tables from db1. Before we do anything, let's see where we stand:
$ psql -U readonly db1 db1=> select * from bar1; ERROR: permission denied for relation bar1 Time: 1.043 ms
One of the first things you noticed (hopefully) is our readonly user could connect to db1 without doing anything, this is because users have CONNECT privileges by default, but even though readonly could connect to db1, we couldn't look at the table because we hadn't given readonly any privileges to do so.
OK, so we better dig into the manual and learn all about the GRANT query, and since we've already spent some time learning about schemas previously, we understand each of our databases has a separate public schema, and those schemas contain our tables, so we probably want to grant some permissions to readonly in our public schema, something like:
$ sudo -u postgres psql -d db1 -c "GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly" $ sudo -u postgres psql -d db1 -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly"
Let's test it out:
$ psql -U readonly db1 db1=> select * from bar1; (No rows) Time: 0.758 ms db1=> \c db2 You are now connected to database "db2" as user "readonly". db2=> select * from bar2; ERROR: permission denied for relation bar2 Time: 0.939 ms
It looks like it worked, but just to be sure, let's double check, first, we'll try and create a new table:
$ psql -U readonly db1 db1=> CREATE TABLE bar3 (_id INT); CREATE TABLE Time: 6.279 ms db1=>
Oh snap! That shouldn't have worked, what the heck? Well, let's dig into the permissions of the public schema:
$ psql -U readonly db1 db1=> \dn+ List of schemas -[ RECORD 1 ]-----+----------------------- Name | public Owner | postgres Access privileges | postgres=UC/postgres | =UC/postgres Description | standard public schema
What does that
=UC/postgres mean? I'm glad you asked, the answer can be found, of course, in the manual:
rolename=xxxx -- privileges granted to a role =xxxx -- privileges granted to PUBLIC U -- USAGE C -- CREATE /yyyy -- role that granted this privilege
Huh, the public schema has CREATE privileges by default, well that was unexpected. So we need to remove those privileges from the readonly user, should be easy enough using a revoke query:
$ sudo -u postgres psql db1 psql (9.3.14) Type "help" for help. db1=# REVOKE CREATE ON SCHEMA public FROM readonly; REVOKE Time: 3.782 ms db1=#
Easy peasy, let's try again:
$ psql -U readonly db1 db1=> CREATE TABLE bar4 (_id INT); CREATE TABLE Time: 7.259 ms
Turns out, revoking a privilege from the user directly doesn't override the granted privileges on public. Why? I don't know:
@user3669651: But it's not the same. Revoking privileges from readonly2 doesn't change the fact that every user can create tables in the public schema until you revoke from role public ... – Erwin Brandstetter Jun 20 '14 at 23:32
It really doesn't make any sense to me, but this is the reality we have to deal with, and we need our readonly user to not be able to create tables, so we have to change up the public schema:
$ sudo -u postgres psql -d db1 -c "REVOKE CREATE ON SCHEMA public FROM public"
And, of course, we'll want to give all the permissions back to our owner, because if we don't, they won't be able to create tables anymore (go ahead, ask me the fun way I discovered this):
$ sudo -u postgres psql -d db1 -c "GRANT ALL ON schema public TO foo"
Now, our readonly user shouldn't be able to create tables anymore:
$ psql -U readonly db1 db1=> CREATE TABLE bar5 (_id INT); ERROR: permission denied for schema public Time: 1.082 ms
Boom, we're finally making progress! I'm sure it's just a formality, but let's make sure we can read new tables also:
$ psql -U foo db1 db1=> CREATE TABLE bar5 (_id INT); CREATE TABLE Time: 7.252 ms db1=> \q $ psql -U readonly db1 db1=> select * from bar5; ERROR: permission denied for relation bar5 Time: 0.950 ms
And we were doing so well.
So now we'll need to spend some more time researching, where we'll eventually realize that granting permissions only works for things that existed the moment those permissions were granted, but if we wanted to have those permissions moving forward for all the new things, we'll need to alter the default privileges that are applied when those new things are created, so let's do that:
$ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT, USAGE ON SEQUENCES TO readonly" $ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT ON TABLES TO readonly"
These are very similar to our original GRANT queries (which feels like we executed a lifetime ago now).
Let's make sure the queries worked:
$ psql -U foo db1 db1=> CREATE TABLE bar6 (_id INT); CREATE TABLE Time: 5.025 ms db1=> \q $ psql -U readonly db1 db1=> select * from bar6; (No rows) Time: 0.816 ms
Well what do you know, success!
The most important part of those ALTER DEFAULT commands is the
...FOR USER foo..., if you don't have that part they won't work (which would lead to quite a while trying to figure out why the query does nothing, not that I would know), and it's also important to run those queries for any user that is going to be creating stuff in that database, because:
$ sudo -u postgres psql db1 db1=# CREATE TABLE bar7 (_id INT); CREATE TABLE Time: 5.430 ms db1=# \q $ psql -U readonly db1 db1=> select * from bar7; ERROR: permission denied for relation bar7 Time: 0.920 ms db1=> \q
So you would need to run the ALTER DEFAULT queries with
...FOR USER postgres... in order for the readonly user to be able to read any tables created in db1 by the postgres user.
Well, we made it, after all that, I feel like we've really grown as people, and as friends. Hopefully this little trip into the abyss that is Postgres's permissions has been helpful :)
There is so so so much more that could be talked about, but in the interest of simplicity I've assumed you would use the default public schema and that you don't care about things like search paths. ↩
Unless you read this guide, then I've got you covered! ↩
These are my slides/notes from a First Opinion company all hands presentation I gave in February 2016
What do you think of when you think of constraints?
Maybe you think of this?
I think of creativity, because constraints breed creativity.
Look no further than music
LEGOS!!!!! Simple lego bricks like this…
Give us things like this...
And, of course, because we’re in San Francisco, I couldn't resist this last example.
Any other examples you can think of where constraints breed creativity?
So what happens when you don't have constraints?
Star Wars Episode 1 - The Phantom Menace. The very definition of no constraints. George Lucas had complete control, from script and casting all the way down the line to post processing.
There are no constraints on the human mind, no walls around the human spirit, no barriers to our progress except those we ourselves erect.
So why do we set deadlines? And pare down feature sets? And don't just hire more developers and designers and product people? Because we are, to some extent, erecting constraints to increase our creativity and productivity as a company.
In the 1980s and 1990s, Royal Farros was the vice president of development for T/ Maker, a small but influential software company. He says, "A lot of us set deadlines that we knew were impossible, enough so to qualify for one of those Parkinson's Law corollaries. 'The time it will take to finish a programming project is twice as long as the time you've allotted for it.' I had a strong belief that if you set a deadline for, say, six months, it would take a year. So, if you had to have something in two years, set the deadline for one year. Bonehead sandbagging, but it always worked."
Below is an excerpt from when we launched First Opinion
We released our first real public version to the app store right before Thanksgiving, which in retrospect might not have been the best idea since most of us left on vacation immediately after. McKay wanted to be very hands on with the matching in the first release. ... So when a new user signed up, McKay would get a notification, he would look over their details and decide which doctor would be right for them.
And since I was going to be visiting my wife's family, I wanted to make sure those matching notifications were rock solid, because if McKay wasn't getting notified, the user wasn't getting matched with a doctor. So I rigged the server to send an email, a text message, and a push notification for each new user that signed up.
Over the next couple of days, First Opinion steadily climbed the app store rankings, moving into the top five apps in the medical category, and McKay's phone blew up with notifications, three at a time, to the point where he couldn't get any sleep because his phone was buzzing every few minutes ... On the flip side, each of our doctors was getting inundated with tons of new users every hour, all with a question or two to ask.
During this time, we were operating under the gun, as we were working to reconcile our growth with the amount of doctors we didn't have (we launched with only a few doctors but gained thousands of users in those first couple of days).
But as we scrambled to handle the load we figured out some incredible features of our Doctor application that helped our doctors manage the load, these are features we still use to this day, but they were created while we were working under intense pressure (which is a constraint) to handle our user load with the few doctors we had at the time.
He told me a story of how Larry Ellison actually got efficiencies from teams. If a team wasn't productive, he'd come every couple of weeks and say, "let me help you out." What did he do? He took away another person until the team started shipping…
I'm Closing with this quote because I think it perfectly encapsulates why we need constraints.
A few months after I gave this presentation, this tweet came through my stream, which I think is relevant...
Road Runner and Coyote
- The Road Runner cannot harm the Coyote except by going "meep, meep."
- No outside force can harm the Coyote -- only his own ineptitude or the failure of Acme products. Trains and trucks were the exception from time to time.
- The Coyote could stop anytime -- if he were not a fanatic.
- No dialogue ever, except "meep, meep" and yowling in pain.
- The Road Runner must stay on the road -- for no other reason than that he's a roadrunner.
- All action must be confined to the natural environment of the two characters -- the southwest American desert.
- All tools, weapons, or mechanical conveniences must be obtained from the Acme Corporation.
- Whenever possible, make gravity the Coyote's greatest enemy.
- The Coyote is always more humiliated than harmed by his failures.
- The audience's sympathy must remain with the Coyote.
- The Coyote is not allowed to catch or eat the Road Runner.
And while the rules might not have actually existed, the cartoons--of which there are 48 shorts, a half-hour special, and one full length movie--follow them pretty closely. 11 rules, ~450 minutes of entertainment.
One of my coworkers sent me this article after I gave this presentation, one library decided to stop sending another library papyrus, so the other library invented parchment (emphasis mine):
One of the Ptolemies’ most drastic schemes to strike down the Library of Pergamum was the sudden cut of its trade of papyrus with the city of Pergamon. The Ptolemies hoped that if the main component of books was limited and hard to obtain, it would prevent the Library of Pergamum’s collection from growing. However, Pergamon came up with an alternative. Roman writer and scholar Marcus Terrentius Varro documented the event: “the rivalry about libraries between king Ptolemy and king Eumenes, Ptolemy stopped the export of papyrus … and so the Pergamenes invented parchment.”
While it’s not possible for Pergamon to have invented parchment since scriptures on stretched leather have been found earlier in the east, the lack of papyrus may have pushed the king to expand the use and development of leather as a writing material, Coqueugniot says. The word for parchment in Latin, “pergamīnum” literally translates to “the sheets of Pergamum,” she says.
This feels like something about constraints but I'm not sure what, I'm including it though because why not? And the quote has a fun attribution history:
I wrote you a long letter because I didn't have time to write you a short letter
My favorite alternative quote is the Woodrow Wilson one on giving speeches from 1918:
“That depends on the length of the speech,” answered the President. “If it is a ten-minute speech it takes me all of two weeks to prepare it; if it is a half-hour speech it takes me a week; if I can talk as long as I want to it requires no preparation at all. I am ready now.”
From The Last Don:
"I've never understood how I can have ten percent of the profit of a picture that grosses one hundred million dollars and costs only fifteen million to make, and then never see a penny. that's one mystery I'd like to solve before I die."
"It's absolutely legal," she said. "They are abiding by the contract, one you should not have signed in the first place. Look, take the one-hundred-million gross. The theaters, the exhibitors, take half, so now the studio only gets fifty million, which is called the rentals."
"OK. The studio takes out the fifteen million dollars the picture costs. Now there's thirty-five million left. But by the terms of your contract and most studio contracts, the studio takes thirty percent of the rentals for distribution costs on the film. That's another fifteen mil in their pockets. So you're down to twenty mil. Then they deduct the cost of making prints, the cost for advertising the picture, which could easily be another five. You're down to fifteen. Now here's the beauty. By contract, the studio gets twenty-five percent of the budget for studio overhead, telephone bills, electricity, use of sound stages etc. Now you're down to eleven million. But the Bankable Star gets at least five percent of the rentals, the director and producer another five percent. So that comes to another five million. You're down to six million. At last you'll get something. But not so fast. They then charge you all the costs of distribution, they charge fifty grand for delivering the prints to the English market, another fifty to France or Germany. And then finally they charge the interest on the fifteen million they borrowed to make the picture. And there they lose me. But that last six million disappears."
Remember people, always negotiate for a percentage of gross, not net.
When I was a kid, Nintendo was the king of video game consoles, no one, not even Sega, could touch 'em. If you would've told me that within a decade Nintendo would be the third ranked console game maker and Sega wouldn't even be in the fight I would've called you a flat out liar, right there to your face.
It was inconceivable that Nintendo could fall so far, so fast. And now, while my daughter and I sometimes enjoy a mean game of Mario Kart Wii, chances are high my children will grow up never asking for Nintendo anything for Christmas or birthdays, something that a kid from my generation couldn't even imagine1.
I bring up this example because there seems to be a pervasive thought among people I know that it's too late to start something, every idea has already been done to death, every thought written down. So let's see if that really is the case, shall we?
No Way I could build a billion dollar company
You might be interested to know that the Fortune 500 turns over about 30 companies annually. Here's the top 10 on the list over three different periods.
While there are some companies with staying power, there are also new additions. In fact, this is entirely normal:
Because no company, no matter how successful, lasts forever, and because only a fraction of companies survive more than a few decades, turnover of varying degrees is entirely natural.
Because this ebb and flow of new companies onto, and off, the list is a natural occurrence, it means there is always time to start a new company.
It's impossible to make a living as an Author
This one is near and dear to my heart, not just because it's an inside joke in my family, but because my brother-in-law actually just published his first novel. Let's take a look at the best selling authors over three different periods2.
|1||The Chamber by John Grisham||The Da Vinci Code by Dan Brown||Fifty Shades of Grey by E.L. James|
|2||Debt of Honor by Tom Clancy||The Five People You Meet in Heaven by Mitch Albom||The Hunger Games by Suzanne Collins|
|3||The Celestine Prophecy by James Redfield||The Last Juror by John Grisham||Fifty Shades Darker by E.L. James|
|4||The Gift by Danielle Steel||Glorious Appearing by Jerry B. Jenkins and Tim LaHaye||Fifty Shades Freed by E.L. James|
|5||Insomnia by Stephen King||Angels & Demons by Dan Brown||Catching Fire by Suzanne Collins|
|6||Politically Correct Bedtime Stories by James Finn Garner||State of Fear by Michael Crichton||Mockingjay by Suzanne Collins|
|7||Wings by Danielle Steel||London Bridges by James Patterson||Diary of a Wimpy Kid: The Third Wheel by Jeff Kinney|
|8||Accident by Danielle Steel||Trace by Patricia Cornwell||Fifty Shades Trilogy Box Set by E.L. James|
|9||Disclosure by Michael Crichton||The Rule of Four by Ian Caldwell and Dustin Thomason||The Mark of Athena by Rick Riordan|
|10||Remember Me by Mary Higgins Clark||The Da Vinci Code: Special Illustrated Collector's Edition by Dan Brown||Gone Girl by Gillian Flynn|
As you can see, there is a ton of turnover on those lists3, which means there's plenty of room for your great american novel. So you know, get to work!
I could never build a best selling iOS app
|Rank||2009 paid||2009 free||2012 paid||2012 free|
|1||Crash Bandicoot Nitro Kart 3D||Angry Birds||Clash of Clans|
|2||Koi Pond||Google Earth||Doodle Jump||Skype|
|3||Enigmo||Pandora Radio||Fruit Ninja||eBay|
|4||Bejeweled 2 + Blitz||Tap Tap Revenge Classic||Angry Birds Seasons||Google Earth|
|5||iBeer||Shazam||Cut the Rope||Google Search|
|6||Moto Chaser||PAC-MAN Lite||TuneIn Radio Pro|
|8||Flick Fishing||Touch Hockey||Angry Birds Rio||Bump|
|9||Tetris||Labyrinth Lite Edition||FatBooth||BBC News|
|10||Texas Hold'em||Flashlight||Flight Control||Shazam|
Angry birds pretty much dominated 2012, but if you go check the overall top app lists right now (don't worry, I'll wait) you'll see Angry Birds is no where to be found in the top 10 anymore.
So what's your point?
Take any other industry or segment and you will likely find similar churn5. This isn't to say your idea will rise to the top, or that it will be easy. On the contrary, it will take a tremendous amount of work, lot's of concentrated effort, and a little bit of luck (the actual luck and the self made kind). This was just to show you that nothing is set in stone, and things do change, in fact, they seem to be changing faster than ever. And as my wife so succinctly put it, "there's always room for the next big thing."
I asked for a Nintendo every year for years, then a Super Nintendo. ↩
And seriously, was I the only one that had no idea 50 Shades of Grey had more than one book? ↩
This was the hardest data to find, and the problem with it is I think things like Facebook drop off the list because everyone downloads them once and has them on their phone forever more. ↩
I also compiled a list of best selling albums, but I'm sick of making tables. ↩
You notice a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are.
You start small, a simplified interface that emphasizes function over form, no ads to annoy your users and a liberal use policy that makes it easy for your users to fit your product to their needs. People start talking about you, and more importantly, they start using your product. A small but growing group of users can't get enough and start telling their friends about how great your product is over all the other products in the market.
Every month more users keep coming, favoring the ease of use and no hassle simplicity of your product over your competition, not to mention the lack of ads your product currently has. You hire more people, you add more servers, and people just keep coming, they love your product.
After a while, you're huge, you're now the main product on the block and all your competition has faded away into obscurity and also-rans. Some Venture Capitalists take notice and approach you to see if you can turn your product into a real business.
You believe what they are selling you and raise a huge round of funding. You're different, they say, you're special. Sure, your product is free and commoditized, but you've built an audience. You're going to transform all those eyeballs into revenue.
In order to do that though you'll have to make a few changes. You remove a feature here, tweak a policy there. All necessary changes in order to lock down your product for advertisers. A small group of users start to complain, but you don't pay attention. Your changes are working, you're monetizing your audience. You knew some would leave, but everything is going according to plan.
Fast forward an unspecified time in the future. Some random person notices a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are...
I remember a conversation with my wife about a year after graduating, we were living in Washington, D.C. at the time and I was working as a Patent Examiner, with a plan to attend law school the following year, I'll be kind and just say that I didn't particular enjoy reading patents all day, and I wasn't particularly looking forward to law school. My wife was working at a law firm and was commenting about how unhappy all the lawyers at her firm were. She said something I've never forgotten, "I don't want you to come home everyday unhappy and talking about how much you hate your job."
We both quit our jobs a few months later and moved back west, I wanted to try my hand at starting a company because what I really wanted was the flexibility and joy that comes from building something from nothing. And what my wife really wanted was for me to come home from work happy each and every day1.
I bring this up because I've been recently working through some old hard drives that are just full of all kinds of things from the last twelve or so years of my life. One of those old documents contained an employment survey I had filled out in the distant past, the Career Consultation section was interesting.
While I'm embarrassed about a few of those answers, I think it's interesting what I focused on when asked what is most important to me, where I wanted to be in a few years, and what my ideal job was. The only thing that mattered to me was to do something I enjoyed and was exciting to me.
Even back then, I wasn't motivated by money or prestige, I was motivated by the desire to enjoy what I do, because when you spend so much time at one place, and there are going to be so many bad things about any job, you need to really enjoy it when it's good.
Hopefully, everybody will strive for something better than just not hating your job, a good test to see where your true feelings lie about your current employment comes from Senator Schumar:
"I have a little test, when you wake up monday morning, do you feel in the pit of your stomach that you want to go to work. And if you can say yes to that, you're in great shape."
I hope all of you are in great shape :)
Man I love my wife ↩
I think making something inconvenient to use unless they pay you is the absolute worse way to make money. Want to watch that movie on any device? Tough luck! Want to read that article? Well, you can't unless your a subscriber. Oh, you didn't think we would let you listen to that song anywhere you wanted, did you?
I would like to say the idea of making money through inconvenience is the exclusive domain of large dying giants like the RIAA, MPAA, and publishing conglomerates, but it's not, lots of startups fall into this same trap also.
Heck, we even tried inconvenience as a business model at First Opinion. We want First Opinion to be the first place people turn to when they have a medical question, and yet, what did we do? We limited your interaction with our doctors to once a month unless you gave us money. A move that all but assured we wouldn't be the first place you turned to when you had a medical question.
While we gathered extensive stats that told us this model just wasn't working for us, it really hit home as a problem when my wife, my own beautiful wonderful wife, said she had two more days until her free question renewed when I asked her why she wasn't asking her First Opinion doctor a medical question she had.
Our mistake was we were making people stop and think about what they should do, as Influence: The Psychology of Persuasion notes, this is a huge no no:
The renowned British philosopher Alfred North Whitehead recognized this inescapable quality of modern life when he asserted that "civilization advances by extending the number of operations we can perform without thinking about them."
Evan Williams, of Blogger, Twitter, and Medium fame, expanded on this in an XOXO talk he gave in 2013:
"The internet makes human desires more easily attainable. In other words, it offers convenience [and] Convenience on the internet is basically achieved by two things: speed, and cognitive ease [...] If you study what the really big things on the internet are, you realize they are masters at making things fast and not making people think.
"Here's the formula if you want to build a billion-dollar internet company [...] Take a human desire, preferably one that has been around for a really long time [...] Identify that desire and use modern technology to take out steps."
In other words, people would much rather pay for convenience over inconvenience. So, if you're trying to decide what business model to pursue, I would encourage you to choose making your product more convenient, even if it results in less money at the start, because chances are, doubling down on convenience will make you larger in the long run than a model of inconvenience ever would.
This is exactly what we've observed at First Opinion, we dropped our inconvenience model and made access to a First Opinion doctor free for everyone, anytime, and instead decided to focus on charging for additional conveniences like speed of our doctor's responses and the ability to send photos to our doctors.
And if you're a large dying giant that has decided to go the inconvenience path, remember this observation by Clayton Christensen and hopefully take heed:
In nearly every instance of disruption we have studied, the survival instincts of the disruptees—the prior industry leaders who are being disrupted—set in motion defensive actions intended to slow the pace of disruption. In the end, however, the advantages that disruptive competitors bring to customers in terms of quality, cost, convenience, and accessibility become so apparent that the regulations are removed and the disruption proceeds apace.
My sister has had a few home births. I've got a niece that is basically in and out of the hospital in less than a day with new baby in hand. Heck, there was even a couple that used First Opinion to deliver their baby in the backseat of their car.
Unfortunately, easy baby arrivals doesn't seem to be how my wife, Dee, rolls. Our daughter was minutes from being taken by C-Section when she finally arrived, and so I guess I shouldn't have been so surprised when a routine doctor visit a few weeks before the due date went too long1.
Wednesday, July 16, 2014, 5:04pm to 5:14pm
Me: Just curious where you're at?
Dee: Still at the doctor. I think everything is fine but it's a long story. Hopefully coming home soon.
Dee: Don't freak out. I think it's ok
Me: Too late, I'm freaked, you've been at the doctor for over 2 hours
Dee: It's really not as bad as it seems. Just too long to tell you over text. It does look like the baby is breech though. Or at least turned funny
Dee: Doesn't mean he's not healthy or that it won't be ok though.
Me: :( let me know when you're headed home
Now I want you to close your eyes and imagine the doctor just told you the baby is breech and the umbilical chord is hanging down so if you went into labor outside a hospital you would most likely lose the baby. Now, imagine the doctor wants you to go the hospital to have them attempt something called a version, which you've never heard of, where they attempt to re-orient the baby into the correct position while still in the womb. Finally, imagine you had planned to have the baby in Redwood City, you knew that, your husband knew that, and your doctor knew that. Okay, so you need to go to the hospital, what hospital do you go to?
If you're my wife and me? You pile into the car, with makeshift overnight bags, and drive to Redwood City, walk in, go up to Labor and Delivery, where there is only a door that says Stop! No Entry and then back down to admitting, where the helpful clerk tells you you need to go back up to the floor you were just on and through the door that says Stop! No Entry and then down the hall and to the left where the nondescript unmarked door that has a silver buzzer hides Labor and Delivery.
The nurses were not expecting us, which is a seriously bad sign in a situation like this. After some back and forth, we were informed that versions are only performed at the San Francisco hospital, and after a quick phone call it was confirmed that San Francisco was wondering where the heck we were. We would've had a good laugh about the misunderstanding if any of us had been in a laughing mood, maybe someday.
My sister met us at Redwood City and took our daughter, Kenzie, home with her while Dee and I began the long rush hour drive back up to San Francisco. We finally arrived at the correct hospital a little before 9pm, where we were educated about what was about to happen. The doctors would attempt to turn our baby, if that failed, they would prep Dee for a C-Section and put her under anesthesia, and then try again more forcefully to turn him.
If either of those were successful, no C-Section for us. We were also informed versions have only about a 50 percent success rate. Also, they can't use the medication they normally use for the procedure because it elevates heart rate and Dee's heart rate was already a little too high, so our success rate was even lower.
Wednesday, July 16, 2014, 10:23pm
Me: Version attempt 1 will take place soon
My Sister: Good luck and fingers crossed
At about 10:30pm the doctors began the first version attempt. My sister tells this great story about a massage she got in India once2, where she sat in the middle of a room, cross-legged and naked, while someone poured oil over her head and then basically punched her back. She called it the worst massage any person could ever get ever, a version is kind of like that. I pressed my forehead to Dee's forehead while she grunted through the pain of two doctors forcibly pushing on her stomach trying to knock our baby back into the correct position.
Wednesday, July 16, 2014, 10:55pm
Me: Wow, they were able to turn the baby in one of the most uncomfortable massages I've ever witnessed.
My Sister: I can only imagine. Will they monitor through the night?
Me: Now they are going to watch for a few hours to see if she will continue to stay in labor or will return to normal
My Parents: Wonderful! But sorry it was hard.
Me: The doc says he might be inclined to induce since she is one day short of 38 weeks
Me: He's not sure he wants to risk the baby turning again
My Sister: Makes sense.
My Parents: How is dee doing? Give her our love.
Me: She says she doesn't know how she's doing, it was just supposed to be a normal checkup
The next 13 hours are standard labor and delivery fair, if this was a movie, here is where the montage would be. Dee's water broke around 5:49am, they put an epidural in around 7am and prepped the delivery table. They moved Dee into position at about 10:30am, and Hayden was born about 11:44am, an 8 lb 10 oz exciting new addition to our family. Between events I was able to grab a little bit of sleep, and Dee was not.
Thursday, July 17, 2014, 11:44am
Me: Hayden j Marcyes, no idea weight or time but he's here
My Parents: We are so happy. Now you can both get some rest.
My Other Sister: So happy all is well!!!
Me: 8 lb 10 oz
My Other Sister: That's pretty big. Holy cow.
My Parents: Good thing he didn't wait two more weeks
My Sister: Big boy...he looks great. I am tearing up.
That makes two, count them, two narrow C-Section dodges. And two freaking awesome kids.