These are my slides/notes from a presentation I gave at Boom Startup in Utah in July 2011. I found this while cleaning up some old stuff in Dropbox and I figured I would make it public because why not?
Hello, my name is Jay, and I work at Plancast.
Startups are a wicked problem, a wicked problem is defined in Code Complete1 as something you have to solve first to know how to solve it.
Horst Rittel and Melvin Webber defined a "wicked" problem as one that could be clearly defined only by solving it, or by solving part of it (1973). This paradox implies, essentially, that you have to "solve" the problem once in order to clearly define it and then solve it again to create a solution that works. This process has been motherhood and apple pie in software development for decades (Peters and Tripp 1976)...
One of the main differences between programs you develop in school and those you develop as a professional is that the design problems solved by school programs are rarely, if ever, wicked. Programming assignments in school are devised to move you in a beeline from beginning to end. You'd probably want to tar and feather a teacher who gave you a programming assignment, then changed the assignment as soon as you finished the design, and then changed it again just as you were about to turn in the completed program. But that very process is an everyday reality in professional programming.
The only way to know if a startup will work is to try it and find out.
Markets that do not exists cannot be analyzed: Suppliers and customers must discover them together. Not only are the market applications for disruptive technologies unknown at the time of their development, they are unknowable. The strategies and plans that managers formulate for confronting disruptive technological change, therefore, should be plans for learning and discovery rather than plans for execution. This is an important point to understand, because managers who believe they know a market's future will plan and invest very differently from those who recognize the uncertainties of a developing market."
I think it was Steve Blank who said a startup is a company formed to find a business model.
No One Size fits all...not for building your company or, on the technical side, solving scaling issues.
This is the first Plancast Techcrunch traffic spike.
Plancast's database Munin CPU graph, trying to figure out a rogue query right before SXSW.
Funny story, the rogue query was a SELECT * FROM users that was buried in a certain page load and had existed from Plancast's inception. The query wasn't a resource drain in the beginning (small user base) and so it sat there...waiting. I never noticed it because it blended in with the normal server load increases that come from a growing user base, but when we hit a critical mass of users it all of a sudden appeared, literally like flicking a switch, and became an exponential resource hog that kept taking the site down.
I said there is no one size fits all, but now I'm going to retract that a bit and say there are certain things that are pretty much universally needed across all companies.
He told me a story of how Larry Ellison actually got efficiencies from teams. If a team wasn't productive, he'd come every couple of weeks and say "let me help you out." What did he do? He took away another person until the team started shipping and stopped having unproductive meetings. (via)
We suck at hiring, so I've got nothing to say about that6.
Never too early to start defining your company culture.
Ours is helicopters. And I'm not a morning person.
Choose wisely! All of our downtime for the last three months has been because of MongoDB7.
I have absolutely no problems annoying the one or two pedantic people who care about this for the benefit of thousands who don't.
Time for an old-school analogy question: Silicon Valley is to entreprenuers as blank is to actors?
Hoooooray for Silicon Valleywood!
I can't say where a startup should be located, I can say it is cool to be at a party and look over and see the Twitter founders. Or to hang out after some dinner event and listen to Prominent Venture Capitalist talk about raising money for his fund. There is just no other place in the world where stuff like that happens every. Single. Night8.
Any questions at all?
Code Complete, section 5.1, Design Is a Wicked Problem. ↩
Plancast was never larger than about four people, but the people we did hire were amazing, so maybe suck was too strong a word, it might be better phrased as we were incredibly slow to hire. ↩
Here's a fun game: guess what database Path had the most problems with? ↩
I still think this is true and still chose to leave Silicon Valley because there are many other factors that go into choosing where to live, like how good the schools are, or if you want to own your own home, or you like seasons. Also, Israel is pretty great also if you want a tech capital on the other side of the world. ↩
This post was originally published in 2012 on the Startup Grind blog, I've cleaned it up, updated it a bit, and republished it here for archival purposes (also I was pretty surprised I hadn't posted it here already)
For a content site1, be it a blog like this one or a social networking site like Twitter. Attracting--and keeping--users is the hardest thing they will ever do.
Getting a ton of users is the hardest part of any business, that being the case, why would you choose a business model that depends on you getting a ton of users? -Ben Thompson
The answer to that question is obvious, you choose this business model because the rewards are ginormous! But when you do choose it, it's the only thing that matters. Scaling? Hiring? Funding? All that is cake compared to getting millions of people to use your product, which is the new baseline for success.
In all my internet travels, I've seen a lot of sites come and go, I've even tried my hand at one or two of my own, and one thing all content sites have in common is how they turn a random visitor into a loyal user--and how they lose that same user later down the road. I call this pattern the law of three:
A user has to visit a site three times in order to become loyal.
A loyal user needs to find three interesting things each visit to stay loyal.
A loyal user is lost if they don't find three interesting things three times in a row.
Simple, right? Yet most entrepreneurs, when they talk about acquiring new users, they talk sign up flows, a/b testing, and viral coefficients. Those are all important, but they should all come after making sure their content is interesting to the users they are trying to attract. Otherwise, no matter how many users they get coming through the door, those users just turn around and go right back out again.
My favorite anecdote about greasing viral coefficients before getting the interesting content part in place (aka, putting the cart before the horse) was a startup that so effectively greased their viral coefficient, using techniques learned from Greg Tseng, that they literally increased their user sign ups from hundreds a day to tens of thousands a day but eventually had to rollback to their original--less successful--funnels because they were losing most of those new users due to a lack of compelling content.
Virality is an amplifier on a product that is already working, you should only focus on it if you are already getting it because it is very hard to manufacture it from zero -Andrew Chen
A founder that’s focused on user acquisition through virality, but can’t quite get the traction they want, should probably go back ... and make sure that the core users are actually happy and engaged. For B2B startups, they often jump the gun and try to scale revenue before really proving that they’re solving a super painful problem, and usage, engagement and retention are good. ... Startups, unfortunately, don’t spend enough time driving engagement and usage, and as a result don’t create enough sustainable value for their users and customers. They see a bit of usage and figure it’s time to rush into the Virality and Revenue stages, hoping that more users will solve all the problems they’re seeing and ignoring. That’s usually not how it works.
What sites people visit to get their content fix is constantly changing (Fun question, how many sites do you visit today that you also visited five years ago?), and each new site that rises up abides by the law of three. Currently, my wife is addicted to Pinterest2 because every time she visits she's flooded with tons of new and interesting pictures. As an aside (because it made me laugh), when I asked my wife why she loves Pinterest so much, she responded:
It makes you feel artistic and crafty just by looking at the pictures even though you didn't actually do anything
For me, the king of the law of three is Ars Technica, I've been a faithful reader for something like two decades3
In contrast, there was a time I couldn't go even a few hours without checking Digg4. Then one day I didn't find anything interesting. After a few more visits with nothing interesting catching my eye, I stopped regularly visiting Digg altogether, and I haven't really been back since. Users only have a certain number of content sites they visit regularly and so once you've lost a loyal user, it's really really hard to get them back. In fact, it might even be harder than getting them in the first place5.
Having trouble getting users to your own site? Step one is figuring out what things are interesting to the people you are trying to attract. After all, what your users get from your site is your core mechanic, and if it isn't up to snuff you'll never attract the audience you so richly deserve.
In 2012 I was using site only for website, now I'm using it as a standin for app or website. ↩
Fun fact, this was from 2012 and in 2017 my wife still visits Pinterest regularly. ↩
I've visited Ars Technica in three different decades, having first visited it in 1999. ↩
Digg was an entirely different site and company when this was originally written. ↩
As exhibit A I give you Twitter, who mentioned they've had over one billion sign-ups, but only a few hundred million active users, despite near constant media attention and exposure, those 700 million users are probably never coming back. ↩
These are my brushed up notes for a presentation I gave during one of First Opinion's all hands meetings in October 2015.
Types of Mistakes
I want to start with a discussion of the three types of mistakes, as relayed to me by a good friend of mine:
Honest mistakes - Everyone falls victim to an honest mistake sometime in their life (I fall victim to them more than most). You accidentally push incomplete code to production or you leave an ad campaign running just a little too long. We're human, these mistakes happen, so we fix them and move on.
Incompetent mistakes - The people that are committing these mistakes are either out of their league, or incompetent. Either way, it's time to cut them loose or give them a different job.
Process mistakes - Most mistakes are because of a lack of process, so I want to spend the majority of my time talking about this type of mistake and how we seek to minimize them (nobody's perfect) at First Opinion.
Minimizing Process Mistakes
Let's start with Engineering, since I happen to know a thing or two about how the engineering team works.
The first line of defense for engineering is our automated tests. Each major piece of our codebase has a decent amount of tests backing it up, as a completely unscientific informal look at the amount of code we have written just to test the codebase, here are some conservative stats on our three biggest software areas1:
Server - 500+ tests comprising around 15,000 lines of code.
iOS - 60+ tests comprising over 10,000 lines of code.
Web-client - Many tests comprising around 5,000 lines of code.
We are constantly adding new tests and making sure the existing tests are still relevant so our Engineers can have that sweet sweet piece of mind that comes from knowing any changes they make to the codebase don't ripple outward like silent chaos ninjas to cause unforeseen bugs elsewhere in our system because without automated tests...
Making even small changes will become increasingly difficult. Eric Evans in Domain-Driven Design: "When complexity gets out of hand, developers can no longer understand the software well enough to change or extend it easily and safely." Facebook [needs] a huge staff to keep up their momentum maintaining a big ball of mud. ... Releases will break things, because you don't understand the relationships well enough to pretend the impact of your changes. ... Next time management or clients try to convince you to move faster and throw quality under the bus, you can say sure, that will work, as long as you can hire 429 engineers to work on our iOS app. via
In fact, our iOS engineers just spent the better part of the last two weeks knocking down the last untested part of the iOS application in an effort to release the most stable app we've ever released2.
Jarid also recently spent a solid week figuring out how to automate the testing of a new feature we're working on3. I guarantee our competitors didn't devote that much time to ensuring they could automatically test that feature, that's what makes us different.
But wait, there's more. The Engineers also do code reviews, these are when one engineer has another engineer look over their code and make suggestions on how it can be improved. These are incredibly valuable in helping all the engineers understand the codebases and helps the codebase stay high quality and well documented since your best critic isn't yourself, it's your colleague who has to maintain your bug infested code.
We try and automate all the things. When we deploy, we deploy with one simple command. When we add servers to our system, that's also one command. Automation means we do things the same way each and every time, thus minimizing mistakes.
When we do find mistakes, we fix the automation scripts, for a change once, fix everywhere workflow.
We have a full time user researcher that is constantly going out and talking to people about our product and planned upcoming features. Think of that for a minute, we have someone whose whole job is to go out and speak to real people about our product, do you know any other company that does that?
Before any major feature gets anywhere near engineering, it's gone through multiple rounds of user testing using paper wireframes, Then onto small little interactive prototypes on the phone, and only after all that is the feature ready to be passed to engineering to be built.
On the doctor side
Each of our main doctors has a staff of support doctors that help them out, our matched doctors review the conversations and monitor the quality of the interaction each one of our users has on our service.
We also have a full time QA staff that works continuously to make sure each and every interaction with a Doctor on our app is a high quality one.
Besides that, we ask the user's themselves to rate their interaction with our doctors and take that feedback incredibly seriously.
But things always go wrong
It's true, they do, and we're no exception. So when a user does have a bad interaction, we kill them with kindness, and we work with them personally to make sure their issue is resolved to their satisfaction.
"Reverse engineer your successes and turn best practices into best processes"
Whenever we have problems, we go through what caused the problem and talk about how we fixed it and what we are going to do different to make sure we never see the problem again, this is the best way to make sure we are always solving new problems and moving forward, instead of going insane.
In Engineering, we also change the code, add tests, or automate the problem away, the goal of these post-mortems and the implementation of the solutions is to minimize the same mistake happening a second time.
We also do one more thing...
We pass it on down
Anyone familiar with the Banana story?
Start with a cage containing five monkeys. Inside the cage, hang a banana on a string and place a set of stairs under it. Before long, a monkey will go to the stairs and start to climb towards the banana. As soon as he touches the stairs, spray all of the other monkeys with cold water. After a while, another monkey makes an attempt with the same result - all the other monkeys are sprayed with cold water. Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.
Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and wants to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted.
Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm! Likewise, replace a third original monkey with a new one, then a fourth, then the fifth.
Every time the newest monkey takes to the stairs, he is attacked. Most of the monkeys that are beating him have no idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey.
After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water. Nevertheless, no monkey ever again approaches the stairs to try for the banana.
Because as far as they know that's the way it's always been done around here.
While this story usually has a negative connotation, it doesn't have to be that way, passing down good company culture is a great thing and helps makes us stronger.
We learn from each other, hopefully we retain mostly the good stuff and fix the bad stuff.
This fanatical obsession with scalable quality assurance has been going on from the very first line of code and the very first user/doctor interaction, to where we sit now at hundreds of interactions a day, to the future at thousands of interactions, and tens of thousands of interactions, and millions of interactions4. We focus so much on our internal quality because we know our internal quality eventually becomes our external quality.
This is what makes us different, this is what makes us special, and this is what will help us change healthcare.
This was incredibly unscientific, for example, in the iOS repo, I went into the testing directory and ran: find . -name "*.m" | xargs wc -l to get the line count. ↩
On Tuesday, November 8, 2016, we voted, and Donald Trump will be the next President of the United States.
While watching the coverage of the election over the last year, I couldn't understand how any rational person could ever vote for someone like Trump, but even the most rational of people can work against their best interests when they feel unfairly treated, as demonstrated in an economic experiment called the ultimatum game1:
The first player (the proposer) receives a sum of money and proposes how to divide the sum between the proposer and the other player. The second player (the responder) chooses to either accept or reject this proposal. If the responder accepts, the money is split according to the proposal. If the responder rejects, neither player receives any money.
The responder typically vetos the deal when low offers are made. People consistently prefer getting nothing to receiving a small share of the pie. Rejecting the offer is in effect paying to punish the proposer.
The reason why this experiment is interesting is because in a completely rational world, the responder would accept any amount of money the proposer offers, because anything is better than nothing, right? Not so fast, we let emotion and passion guide us all the time, especially when we think we're getting a raw deal or feeling betrayed, as Ben Thompson wrote in his daily update after the election2:
People feel the system is unfair, that they don’t have a stake in it, and they are exceptionally mad. This is a sentiment that has been rising for decades thanks to globalization and the increase in manufacturing productivity that devastated so much of the Midwest in particular, but it was dramatically accelerated by the aftermath of the Great Recession [of 2008] when, in the estimation of people where I grew up, Wall Street got off scot-free while everyone else had to bear the pain. I cannot overstate how pervasive and deeply-felt this sentiment is: so many people I know — regardless of income level — deeply believe the system is rigged, so who cares if Trump destroys the whole thing?
They're getting the s**t kicked out of them. I know, I was there. Step outside of the city, and the suicide rate among young people f**king doubles. The recession pounded rural communities, but all the recovery went to the cities. The rate of new businesses opening in rural areas has utterly collapsed.
See, rural jobs used to be based around one big local business -- a factory, a coal mine, etc. When it dies, the town dies. Where I grew up, it was an oil refinery closing that did us in. I was raised in the hollowed-out shell of what the town had once been. The roof of our high school leaked when it rained. Cities can make up for the loss of manufacturing jobs with service jobs -- small towns cannot. That model doesn't work below a certain population density.
If you don't live in one of these small towns, you can't understand the hopelessness. The vast majority of possible careers involve moving to the city, and around every city is now a hundred-foot wall called "Cost of Living."
This sense of loss and unfairness has been rising for decades, but it feels like it has only been in the last eight years or so that this unease has reached the tipping point and started to manifest itself politically, Game Change touches on this unrest boiling to the surface during the 2008 elections:
As the election barreled toward its conclusion, something dark and frightening was unleashed, freed in part by the words of the McCains and Palin. At rallies across the country, there were jagged outbursts of rage and accusations of sedition hurled at Obama...
Colin Powell had been friends with McCain for twenty-five years. The senator had been actively seeking his endorsement ... Powell warned McCain that his greatest reservation was the intolerant tone that seemed to be overtaking the Republican Party. McCain’s selection of Palin bothered Powell because he saw her as polarizing ... And then there were the hate-soaked rallies...
The general’s repudiation was a stinging blow for McCain ... Powell represented the same brand of Republicanism as McCain’s. Tough on defense. Fiscally prudent. Pragmatic and nondoctrinaire. McCain had to wonder what had become of him if his current incarnation was repelling someone like Powell. He was startled by the crazies at his rallies. Who were they? Why were they there? And what did they see in him?
And, indeed, the first Tea Party candidates were elected in 2010, marking the beginning of a new era of political unrest.
Just before the election we packed up the kids and headed south from San Francisco to Disneyland. On our way we passed many a barren field with a sign reading, "congress created dust bowl"
And oftentimes, right next to those dustbowl signs, was a "Farmers for Trump" sign:
Signs like this are usually preaching to the choir though, because as it turns out, America votes by population density, more or less
As cities continue to grow in red states, those cities will become more blue, and ultimately, those states will become more purple, and then blue...
If you follow the red state trend lines, you can clearly see that any dense, fast-growing cities that might emerge in red states will be very likely to vote blue. The few that do already exist already vote blue. How would these new cities be different and [what would] cause them to vote red?
Let's try and answer that question: what would cause cities and/or states to vote red? There really are only two answers:
So how would you cause more rural voters to vote and less urban voters to vote? On the rural front, the answer seems to be a growing sense of unfairness, tinged with a fair bit of legitimate anger, and topped with a helping of more rural voters actually bothering to vote, in larger and larger numbers, in each election:
In 2008 and 2012, rural voters accounted for only about 19 percent of the vote in the state, but according to preliminary exit poll results, they account for 27 percent of the state’s vote. And Trump does well with those rural voters, winning them by about 15 points.
Trump leads Democratic rival Hillary Clinton by the commanding margin of 55 percent to 18 percent among farmers with operations of at least 200 acres.
Why would a lifelong urbanite ... enjoy such robust support in the heartland? The answer may lie in The Donald's unmatched ability to deliver an angry, frothing rant. The Agri-Pulse poll finds that 86 percent of respondents reported being somewhat or very dissatisfied with the way things are going in the United States...
As Siena Chrisman recently put it on Civil Eats, "Rural America is mad." She notes that Trump polls strongest in places where "white identity mixes with long-simmering economic dysfunctions," quoting a New York Times analysis. And few places combine embattled white identity and economic stress quite like farm country.
And how would you get urban voters to not show up? Choose some candidates no one is particularly excited about
In Michigan, where Clinton lost by around 13,000 votes, some analysts estimate that 90,000 Democrats left the top line blank.
Now take the results in Michigan and multiply them across the entire country:
Now let's visualize what that all those blue votes disappearing looks like using some 2008to2016 county maps (red for Republican, blue for Democrat):
Notice anything? The incredibly shrinking number of blue counties by chance? Particularly among the rust belt cities, which have been hit really really hard over the last 8 years and maybe aren't enamored with the current status quo? So now we know, that's how you get previously blue voting counties to become red.
heavily urban counties ... were 32 percentage points more Democratic in their voting than the average. The heavily rural counties were about 11 percentage points more Republican...
If you plot every county's urban-versus-rural divide by the per-election average change in the vote, the pattern is clear: more urban areas vote have been voting more Democratic...
What's particularly interesting is to see how each county has changed over time. Notice that urban areas, counties containing cities, have grown more Democratic; the counties surrounding them (usually suburbs) less so....
the political shifts that have emerged as people of like-minded politics have moved into similar and dense areas ... over time, the tension between rural and urban areas in every state could become significant factors in politics, if this polarization continues.
In the endless expanse of communications the Internet is, probably the greatest and most terrible gift it offers to furries ... is the ability to shut themselves off from the mainstream. They huddle in cloisters that are virtually unassailable by the outside world and whisper encouraging things to one another that would be nearly impossible to say in real life. Free from the pressures of society to conform to a boring standard they go in the exact opposite direction, externalizing things that are roughly as far from "normal" as can be expected. Then, within their protected virtual enclaves, they declare these things to be the norm. By declaring their perversions, mores, and general imbecility to be their own status quo they have simultaneously validated their own existence and demonstrated the inferiority of outsiders.
It seems that people just don’t want to compromise on anything anymore, and it seems to be getting worse and worse with each and every year, people are clustering to their respective side and refusing to venture towards the middle for any reason...
When you can surround yourself with a community that only shares your beliefs, you have a recipe for killing any and all compromise, why should I ever try and see your point of view when all my peeps over here are yelling and screaming that you suck?
in an aggregated world it is voters aka users who decide which issues get traction and which don’t. And, by extension, the most successful politicians in an aggregated world are not those who serve the party but rather those who tell voters what they most want to hear.
It’s tempting to imagine that rising political polarization is just a temporary blip and America will soon return to a calmer, friendlier political system. Don’t bet on it. Political polarization maps onto more than just politics. It’s changing where people live, what they watch, and who they see — and, in all cases, it’s changing those things in ways that lead to more political polarization, particularly among the people who are already most politically polarized…
It’s easy to see how this could work to strengthen polarization over time ... people become more extreme when they’re around others who share their beliefs. If liberals and conservatives end up moving to different places and surrounding themselves with others like them they’re likely to pull yet further apart. And even for those who can’t move, the internet makes it easy to settle in a virtual neighborhood with people who agree with you. Polarization is going to get a lot worse before it starts getting better.
Hillary Clinton had just about every single endorsement from newspapers and magazines across the country – as well as a majority of mainstream TV personalities – on her side, even if reluctantly. And it meant nothing at all.
And not only are people no longer listening to traditional gatekeepers, they are actively preventing established power structures from taking control from them:
Republican voters ultimately did go for Trump, however. In fact, they intervened to wrest control of the nomination back from the delegates. Until just a few weeks ago, Trump’s position had stagnated in national polls and in state-by-state results; before New York on April 19, he’d yet to win a majority in any state, lending credence to the idea that he might have a “ceiling” (albeit a higher ceiling than some originally expected) on his support. From New York onward, however, Trump has won a majority in every state ... with a sudden lunge upward after Wisconsin, rather than a smooth upward projection.
What happened after Wisconsin? My theory ... is that Republican voters were swayed by Trump’s arguments that the candidate with the most votes and delegates should be the nominee ... Some voters might have preferred Cruz or John Kasich to Trump in the abstract, but not at the expense of a contested convention in which the plurality winner would be denied the nomination and replaced with another flawed candidate.
A few weeks back I had the privilege of talking to a well known CEO, as the topic turned to politics, he talked about how Trump would never win, but when I pointed out that the people who voted for him don't actually go away when the election is over, he responded with, "yeah, what are they going to do about it?"3
A few days after the election, while talking to another CEO about the election, he asked, "what are we going to do about it?" Hopefully, this election marks a change of attitude for a lot of people (and a smidgen of empathy would be nice also), and the beginning of something great, because change rarely happens during times of political stability, but can come fast during times of instability, my hope is that any change is is ultimately positive, not just for a privileged few Americans, but for every american, in every state.
In the end, for all Trump's talk about how the election is rigged, I think we've just seen the best possible proof it is not. That ultimately, people will be heard, and how important each person's vote really is.
the included quote is a mashup of this and this Wikipedia page. ↩
When I started writing this post I thought Trump was going to lose, and I was going to use this story as a means of talking about how when people are really angry, what they do is start revolutions: like The American Revolution, and the French Revolution, and the American Civil War. ↩
So we needed to add a Postgres user that could only read tables from our database--not only the currently existing tables, but any tables added in the future--and I found Postgres's permissions system surprisingly complex and unfriendly.
Sure, everything you need to know is buried throughout multiplemanual pages and tonsofStackOverflowanswers, amongother places, but I didn't feel there was anything out there that brought together all the disparate sources into one easy to follow beginner's guide to understanding Postgres permissions, so here we are, the easy guide to creating a permanent read-only users in PostgreSQL1.
Before we dive in, I'm using version 9.3 of Postgres, on Ubuntu 14.04:
$ psql --version
psql (PostgreSQL) 9.3.11
So all this information will apply to that version of Postgres, and all the commands will work on that version of Ubuntu. I'm not sure what changes, if any, would need to be made for earlier, later, or different versions of Postgres and Linux.
It really helps to understand some of the different data structures that come into play when messing with Postgres's permissions, by default, every database you create has a default schema named public, the schema you use is incredibly important and could be a great source of frustration and annoyance2 as you mess with the users and try and set their permissions.
But it won't be frustrating for us because we're going to learn by example, on a fresh install of Postgres, let's create our two users, our owner foo and our read only user readonly:
$ sudo -u postgres psql -c "CREATE USER foo WITH ENCRYPTED PASSWORD '...'" -d template1
$ sudo -u postgres psql -c "CREATE USER readonly WITH ENCRYPTED PASSWORD '...'" -d template1
and let's have our new foo user own our two databases:
So now database db1 should have a bar1 table, and database db2 should have a bar2 table. Now we're ready to understand the public schema of the database.
What we are going to do is connect to db1 and list its tables, then connect to db2 and list its tables:
vagrant@vagrant:~$ psql -U foo db1
List of relations
-[ RECORD 1 ]--
Schema | public
Name | bar1
Type | table
Owner | foo
db1=> \c db2
You are now connected to database "db2" as user "foo".
List of relations
-[ RECORD 1 ]--
Schema | public
Name | bar2
Type | table
Owner | foo
Notice that each table belongs to schema public, but each public schema has a different table. So now we understand that when we mess with the public schema, we need to be cognizant of what database we are connected to because any changes we make to the public schema will only affect the public schema of the connected database.
The second thing we should undertand is Postgres seems to use role, group, and userinterchangeably:
The concept of roles subsumes the concepts of "users" and "groups". In PostgreSQL versions before 8.1, users and groups were distinct kinds of entities, but now there are only roles. Any role can act as a user, a group, or both.
We're only ever going to use user here, but when looking at different sources of information, you might find people using role or, less likely, group3.
The readonly user
We want the readonly user to be able to read all tables from db1. Before we do anything, let's see where we stand:
$ psql -U readonly db1
db1=> select * from bar1;
ERROR: permission denied for relation bar1
Time: 1.043 ms
One of the first things you noticed (hopefully) is our readonly user could connect to db1 without doing anything, this is because users have CONNECT privileges by default, but even though readonly could connect to db1, we couldn't look at the table because we hadn't given readonly any privileges to do so.
OK, so we better dig into the manual and learn all about the GRANT query, and since we've already spent some time learning about schemas previously, we understand each of our databases has a separate public schema, and those schemas contain our tables, so we probably want to grant some permissions to readonly in our public schema, something like:
$ sudo -u postgres psql -d db1 -c "GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly"
$ sudo -u postgres psql -d db1 -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly"
Let's test it out:
$ psql -U readonly db1
db1=> select * from bar1;
Time: 0.758 ms
db1=> \c db2
You are now connected to database "db2" as user "readonly".
db2=> select * from bar2;
ERROR: permission denied for relation bar2
Time: 0.939 ms
It looks like it worked, but just to be sure, let's double check, first, we'll try and create a new table:
Oh snap! That shouldn't have worked, what the heck? Well, let's dig into the permissions of the public schema:
$ psql -U readonly db1
List of schemas
-[ RECORD 1 ]-----+-----------------------
Name | public
Owner | postgres
Access privileges | postgres=UC/postgres
Description | standard public schema
What does that =UC/postgres mean? I'm glad you asked, the answer can be found, of course, in the manual:
rolename=xxxx -- privileges granted to a role
=xxxx -- privileges granted to PUBLIC
U -- USAGE
C -- CREATE
/yyyy -- role that granted this privilege
Huh, the public schema has CREATE privileges by default, well that was unexpected. So we need to remove those privileges from the readonly user, should be easy enough using a revoke query:
$ sudo -u postgres psql db1
Type "help" for help.
db1=# REVOKE CREATE ON SCHEMA public FROM readonly;
Time: 3.782 ms
Turns out, revoking a privilege from the user directly doesn't override the granted privileges on public. Why? I don't know:
@user3669651: But it's not the same. Revoking privileges from readonly2 doesn't change the fact that every user can create tables in the public schema until you revoke from role public ... – Erwin Brandstetter Jun 20 '14 at 23:32
It really doesn't make any sense to me, but this is the reality we have to deal with, and we need our readonly user to not be able to create tables, so we have to change up the public schema:
$ sudo -u postgres psql -d db1 -c "REVOKE CREATE ON SCHEMA public FROM public"
And, of course, we'll want to give all the permissions back to our owner, because if we don't, they won't be able to create tables anymore (go ahead, ask me the fun way I discovered this):
$ sudo -u postgres psql -d db1 -c "GRANT ALL ON schema public TO foo"
Now, our readonly user shouldn't be able to create tables anymore:
$ psql -U readonly db1
db1=> CREATE TABLE bar5 (_id INT);
ERROR: permission denied for schema public
Time: 1.082 ms
Boom, we're finally making progress! I'm sure it's just a formality, but let's make sure we can read new tables also:
$ psql -U foo db1
db1=> CREATE TABLE bar5 (_id INT);
Time: 7.252 ms
$ psql -U readonly db1
db1=> select * from bar5;
ERROR: permission denied for relation bar5
Time: 0.950 ms
And we were doing so well.
So now we'll need to spend some more time researching, where we'll eventually realize that granting permissions only works for things that existed the moment those permissions were granted, but if we wanted to have those permissions moving forward for all the new things, we'll need to alter the default privileges that are applied when those new things are created, so let's do that:
$ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT, USAGE ON SEQUENCES TO readonly"
$ sudo -u postgres psql -d db1 -c "ALTER DEFAULT PRIVILEGES FOR USER foo IN SCHEMA public GRANT SELECT ON TABLES TO readonly"
These are very similar to our original GRANT queries (which feels like we executed a lifetime ago now).
Let's make sure the queries worked:
$ psql -U foo db1
db1=> CREATE TABLE bar6 (_id INT);
Time: 5.025 ms
$ psql -U readonly db1
db1=> select * from bar6;
Time: 0.816 ms
Well what do you know, success!
The most important part of those ALTER DEFAULT commands is the ...FOR USER foo..., if you don't have that part they won't work (which would lead to quite a while trying to figure out why the query does nothing, not that I would know), and it's also important to run those queries for any user that is going to be creating stuff in that database, because:
$ sudo -u postgres psql db1
db1=# CREATE TABLE bar7 (_id INT);
Time: 5.430 ms
$ psql -U readonly db1
db1=> select * from bar7;
ERROR: permission denied for relation bar7
Time: 0.920 ms
So you would need to run the ALTER DEFAULT queries with ...FOR USER postgres... in order for the readonly user to be able to read any tables created in db1 by the postgres user.
Well, we made it, after all that, I feel like we've really grown as people, and as friends. Hopefully this little trip into the abyss that is Postgres's permissions has been helpful :)
There is so so so much more that could be talked about, but in the interest of simplicity I've assumed you would use the default public schema and that you don't care about things like search paths. ↩
Unless you read this guide, then I've got you covered! ↩
There are no constraints on the human mind, no walls around the human spirit, no barriers to our progress except those we ourselves erect.
So why do we set deadlines? And pare down feature sets? And don't just hire more developers and designers and product people? Because we are, to some extent, erecting constraints to increase our creativity and productivity as a company.
In the 1980s and 1990s, Royal Farros was the vice president of development for T/ Maker, a small but influential software company. He says, "A lot of us set deadlines that we knew were impossible, enough so to qualify for one of those Parkinson's Lawcorollaries. 'The time it will take to finish a programming project is twice as long as the time you've allotted for it.' I had a strong belief that if you set a deadline for, say, six months, it would take a year. So, if you had to have something in two years, set the deadline for one year. Bonehead sandbagging, but it always worked."
We released our first real public version to the app store right before Thanksgiving, which in retrospect might not have been the best idea since most of us left on vacation immediately after. McKay wanted to be very hands on with the matching in the first release. ... So when a new user signed up, McKay would get a notification, he would look over their details and decide which doctor would be right for them.
And since I was going to be visiting my wife's family, I wanted to make sure those matching notifications were rock solid, because if McKay wasn't getting notified, the user wasn't getting matched with a doctor. So I rigged the server to send an email, a text message, and a push notification for each new user that signed up.
Over the next couple of days, First Opinion steadily climbed the app store rankings, moving into the top five apps in the medical category, and McKay's phone blew up with notifications, three at a time, to the point where he couldn't get any sleep because his phone was buzzing every few minutes ... On the flip side, each of our doctors was getting inundated with tons of new users every hour, all with a question or two to ask.
During this time, we were operating under the gun, as we were working to reconcile our growth with the amount of doctors we didn't have (we launched with only a few doctors but gained thousands of users in those first couple of days).
But as we scrambled to handle the load we figured out some incredible features of our Doctor application that helped our doctors manage the load, these are features we still use to this day, but they were created while we were working under intense pressure (which is a constraint) to handle our user load with the few doctors we had at the time.
He told me a story of how Larry Ellison actually got efficiencies from teams. If a team wasn't productive, he'd come every couple of weeks and say, "let me help you out." What did he do? He took away another person until the team started shipping…
I'm Closing with this quote because I think it perfectly encapsulates why we need constraints.
A few months after I gave this presentation, this tweet came through my stream, which I think is relevant...
The Road Runner cannot harm the Coyote except by going "meep, meep."
No outside force can harm the Coyote -- only his own ineptitude or the failure of Acme products. Trains and trucks were the exception from time to time.
The Coyote could stop anytime -- if he were not a fanatic.
No dialogue ever, except "meep, meep" and yowling in pain.
The Road Runner must stay on the road -- for no other reason than that he's a roadrunner.
All action must be confined to the natural environment of the two characters -- the southwest American desert.
All tools, weapons, or mechanical conveniences must be obtained from the Acme Corporation.
Whenever possible, make gravity the Coyote's greatest enemy.
The Coyote is always more humiliated than harmed by his failures.
The audience's sympathy must remain with the Coyote.
The Coyote is not allowed to catch or eat the Road Runner.
And while the rules might not have actually existed, the cartoons--of which there are 48 shorts, a half-hour special, and one full length movie--follow them pretty closely. 11 rules, ~450 minutes of entertainment.
One of my coworkers sent me this article after I gave this presentation, one library decided to stop sending another library papyrus, so the other library invented parchment (emphasis mine):
One of the Ptolemies’ most drastic schemes to strike down the Library of Pergamum was the sudden cut of its trade of papyrus with the city of Pergamon. The Ptolemies hoped that if the main component of books was limited and hard to obtain, it would prevent the Library of Pergamum’s collection from growing. However, Pergamon came up with an alternative. Roman writer and scholar Marcus Terrentius Varro documented the event: “the rivalry about libraries between king Ptolemy and king Eumenes, Ptolemy stopped the export of papyrus … and so the Pergamenes invented parchment.”
While it’s not possible for Pergamon to have invented parchment since scriptures on stretched leather have been found earlier in the east, the lack of papyrus may have pushed the king to expand the use and development of leather as a writing material, Coqueugniot says. The word for parchment in Latin, “pergamīnum” literally translates to “the sheets of Pergamum,” she says.
This feels like something about constraints but I'm not sure what, I'm including it though because why not? And the quote has a fun attribution history:
I wrote you a long letter because I didn't have time to write you a short letter
“That depends on the length of the speech,” answered the President. “If it is a ten-minute speech it takes me all of two weeks to prepare it; if it is a half-hour speech it takes me a week; if I can talk as long as I want to it requires no preparation at all. I am ready now.”
"I've never understood how I can have ten percent of the profit of a picture that grosses one hundred million dollars and costs only fifteen million to make, and then never see a penny. that's one mystery I'd like to solve before I die."
"It's absolutely legal," she said. "They are abiding by the contract, one you should not have signed in the first place. Look, take the one-hundred-million gross. The theaters, the exhibitors, take half, so now the studio only gets fifty million, which is called the rentals."
"OK. The studio takes out the fifteen million dollars the picture costs. Now there's thirty-five million left. But by the terms of your contract and most studio contracts, the studio takes thirty percent of the rentals for distribution costs on the film. That's another fifteen mil in their pockets. So you're down to twenty mil. Then they deduct the cost of making prints, the cost for advertising the picture, which could easily be another five. You're down to fifteen. Now here's the beauty. By contract, the studio gets twenty-five percent of the budget for studio overhead, telephone bills, electricity, use of sound stages etc. Now you're down to eleven million. But the Bankable Star gets at least five percent of the rentals, the director and producer another five percent. So that comes to another five million. You're down to six million. At last you'll get something. But not so fast. They then charge you all the costs of distribution, they charge fifty grand for delivering the prints to the English market, another fifty to France or Germany. And then finally they charge the interest on the fifteen million they borrowed to make the picture. And there they lose me. But that last six million disappears."
When I was a kid, Nintendo was the king of video game consoles, no one, not even Sega, could touch 'em. If you would've told me that within a decade Nintendo would be the third ranked console game maker and Sega wouldn't even be in the fight I would've called you a flat out liar, right there to your face.
It was inconceivable that Nintendo could fall so far, so fast. And now, while my daughter and I sometimes enjoy a mean game of Mario Kart Wii, chances are high my children will grow up never asking for Nintendo anything for Christmas or birthdays, something that a kid from my generation couldn't even imagine1.
I bring up this example because there seems to be a pervasive thought among people I know that it's too late to start something, every idea has already been done to death, every thought written down. So let's see if that really is the case, shall we?
Glorious Appearing by Jerry B. Jenkins and Tim LaHaye
Fifty Shades Freed by E.L. James
Insomnia by Stephen King
Angels & Demons by Dan Brown
Catching Fire by Suzanne Collins
Politically Correct Bedtime Stories by James Finn Garner
State of Fear by Michael Crichton
Mockingjay by Suzanne Collins
Wings by Danielle Steel
London Bridges by James Patterson
Diary of a Wimpy Kid: The Third Wheel by Jeff Kinney
Accident by Danielle Steel
Trace by Patricia Cornwell
Fifty Shades Trilogy Box Set by E.L. James
Disclosure by Michael Crichton
The Rule of Four by Ian Caldwell and Dustin Thomason
The Mark of Athena by Rick Riordan
Remember Me by Mary Higgins Clark
The Da Vinci Code: Special Illustrated Collector's Edition by Dan Brown
Gone Girl by Gillian Flynn
As you can see, there is a ton of turnover on those lists3, which means there's plenty of room for your great american novel. So you know, get to work!
I could never build a best selling iOS app
This one is for me, not only because it's my job, but I've even been working on my own app while learning Swift. And it's easy for me to think there is no possibility of my app breaking out, but4...
Crash Bandicoot Nitro Kart 3D
Clash of Clans
Bejeweled 2 + Blitz
Tap Tap Revenge Classic
Angry Birds Seasons
Cut the Rope
TuneIn Radio Pro
Angry Birds Rio
Labyrinth Lite Edition
Angry birds pretty much dominated 2012, but if you go check the overall top app lists right now (don't worry, I'll wait) you'll see Angry Birds is no where to be found in the top 10 anymore.
So what's your point?
Take any other industry or segment and you will likely find similar churn5. This isn't to say your idea will rise to the top, or that it will be easy. On the contrary, it will take a tremendous amount of work, lot's of concentrated effort, and a little bit of luck (the actual luck and the self made kind). This was just to show you that nothing is set in stone, and things do change, in fact, they seem to be changing faster than ever. And as my wife so succinctly put it, "there's always room for the next big thing."
I asked for a Nintendo every year for years, then a Super Nintendo. ↩
You notice a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are.
You start small, a simplified interface that emphasizes function over form, no ads to annoy your users and a liberal use policy that makes it easy for your users to fit your product to their needs. People start talking about you, and more importantly, they start using your product. A small but growing group of users can't get enough and start telling their friends about how great your product is over all the other products in the market.
Every month more users keep coming, favoring the ease of use and no hassle simplicity of your product over your competition, not to mention the lack of ads your product currently has. You hire more people, you add more servers, and people just keep coming, they love your product.
After a while, you're huge, you're now the main product on the block and all your competition has faded away into obscurity and also-rans. Some Venture Capitalists take notice and approach you to see if you can turn your product into a real business.
You believe what they are selling you and raise a huge round of funding. You're different, they say, you're special. Sure, your product is free and commoditized, but you've built an audience. You're going to transform all those eyeballs into revenue.
In order to do that though you'll have to make a few changes. You remove a feature here, tweak a policy there. All necessary changes in order to lock down your product for advertisers. A small group of users start to complain, but you don't pay attention. Your changes are working, you're monetizing your audience. You knew some would leave, but everything is going according to plan.
Fast forward an unspecified time in the future. Some random person notices a gap in a commoditized market, the current market leader isn't satisfying its users and a growing majority are complaining about how hard their product is to use and how annoying the ads are...
I remember a conversation with my wife about a year after graduating, we were living in Washington, D.C. at the time and I was working as a Patent Examiner, with a plan to attend law school the following year, I'll be kind and just say that I didn't particular enjoy reading patents all day, and I wasn't particularly looking forward to law school. My wife was working at a law firm and was commenting about how unhappy all the lawyers at her firm were. She said something I've never forgotten, "I don't want you to come home everyday unhappy and talking about how much you hate your job."
We both quit our jobs a few months later and moved back west, I wanted to try my hand at starting a company because what I really wanted was the flexibility and joy that comes from building something from nothing. And what my wife really wanted was for me to come home from work happy each and every day1.
I bring this up because I've been recently working through some old hard drives that are just full of all kinds of things from the last twelve or so years of my life. One of those old documents contained an employment survey I had filled out in the distant past, the Career Consultation section was interesting.
While I'm embarrassed about a few of those answers, I think it's interesting what I focused on when asked what is most important to me, where I wanted to be in a few years, and what my ideal job was. The only thing that mattered to me was to do something I enjoyed and was exciting to me.
My name is Jay Marcyes, I like to build things, mostly using computers, sometimes using legos. I have three amazing kids, and a very patient wife. I'm currently Cofounder and CTO of First Opinion, previously I cofounded Plancast, which was acquired by Active Network.