Very Joe

The value of information

Thu, 14 May 2020 15:15:00 +0000

How much should you pay for information to help you make a decision?

Imagine that I run a business, and I’m thinking of launching a new product. I think it will sell well, but it’s probably worth running a survey or doing some user research to be more certain. How much should I spend on that research? How much should I value it?

If I’m going to spend $100,000 developing and selling the product, that’s the max I stand to lose, but if I’m already fairly certain it will succeed, wouldn’t it be excessive to spend $100,000 on research?

This post builds upon ideas in the book How To Measure Anything by Douglas W. Hubbard, which I highly recommend. In the book, valuing information is a key part of a larger decision-making framework, but as I read it, I found myself wanting more mathematical justification for some of the statements and ideas.

This post fills in the gaps, by examining the question of how much one should pay for information using a simple gambling game under the lens of maths and probability. We’ll start by figuring out how much you should pay for perfect information on the outcome of the game, then move on to imperfect information, which might be unreliable or even misleading. Finally, we’ll come away with some rules of thumb for thinking about the value of information.

Jump to:

The dice game
Perfect information
Imperfect (misleading) information
Imperfect (useless) information
Rules of thumb

The dice game

I offer to play the following game as many times as you’d like. I’ll roll a dice, and if it lands on 1 or 2, you lose $900. If it lands on 3, 4, 5 or 6, you win $60. Even though you’re more likely to win than lose, the cost of losing is so great that your losses will far outstrip your winnings after a few games.

We can calculate the Expected Value of this game - how much you make or lose on average with each play - by multiplying the value of each outcome with its probability, and adding the results.

Outcome	Probability	Value	Expected Value
1, 2	1/3	-$900	-$300
3, 4, 5, 6	2/3	$60	$40
Overall			-$260

Or as totally legit math notation: first defining the probability mass function of a fair six-sided dice:

\[\Pr(x)_X = \begin{cases} 1/6 & x \in \{1, 2, 3, 4, 5, 6\} \\ 0 & \textrm{otherwise} \\ \end{cases}\]

Then defining the game and computing its expected value as described above:

\[\begin{align} F(x) &= \begin{cases}-900 & x \in \{1, 2\} \\ 60 & x \in \{3, 4, 5, 6\} \end{cases} \\ E[F(x)] &= \sum_{x \in \{1..6\}} \Pr(x)F(x) \\ &= \frac{1}{6} (60 + 60 + 60 + 60 - 900 - 900) \\ &= -260 \end{align}\]

Given that you’ll lose an average of $260 per game, you really shouldn’t want to play it, so we’ll call “don’t play the game” your default decision. If you decide not to play, but I roll the dice to see what would have happened and it lands on a 5, you miss out on $60. That’s Opportunity Loss. Since the chance of that happening is 2/3, the expected value of your opportunity loss, i.e. your Expected Opportunity Loss, is $40.

Here are the same definitions in handy table format:

Term	Definition	Value in this game
Default Decision	The decision you would make given the information you have.	Don’t play!
Opportunity Loss	The amount that you expect to lose (or not make) if your decision was wrong	$60
Expected Opportunity Loss (EOL)	The expected value of Opportunity Loss, i.e. the chance of each loss scenario multiplied by its value.	$60 * 2/3 = $40

Perfect information

Now I’m going to give you the opportunity to learn the result of the game:

I’ll roll the dice secretly,
For a price, I’ll offer to tell you the result,
After buying (or not buying) the result, you’ll decide whether to play.

Because I’m offering you perfect information on the result of this game, you’re never at risk of losing $900. You either:

Don’t buy the result, and don’t play,
Learn that you’ll win and decide to play, or
Learn that you’ll lose, decide not to play.

In this situation, what’s the absolute maximum you should pay to know the dice roll?

Perhaps $60? No - if you pay $60, you’ll make that money back 2/3 of the time, but lose it 1/3 of the time. Overall, you expect to lose $20 per game.
Perhaps $0? That would be great for you, but I’m not gonna give you the information for free!

Let’s solve this with maths. Defining $k$ as the amount paid for perfect information, we want to find the greatest value of $k$ such that the value of playing the game is at least $0. Adding in $k$, the game is defined as:

\[G(x) = \begin{cases}- k + 0 & x \in \{1, 2\} \\ - k + 60 & x \in \{3, 4, 5, 6\} \end{cases}\]

Solving $E[G(x)] \geq 0$ for $k$.

\[\begin{align} 0 &\leq E[G(x)] \\ &\leq \sum_{x \in \{1..6\}} \Pr(x)G(x) \\ &\leq \frac{2}{6}(-k) + \frac{4}{6}(- k + 60)\\ &\leq -k + \frac{4 * 60}{6} \\ \therefore k &\leq40 \end{align}\]

The absolute maximum you should pay is $40, at which point you expect to break even. If you can pay less, you’ll make a profit. If you pay more, you’ll lose in the long run. To demonstrate that, this table shows the expected value of each outcome when you pay $40 for information.

Outcome	Probability	Value	Expected Value
1, 2	1/3	-$40 + $0 = -$40 (decide not to play)	-$13.33
3, 4, 5, 6	2/3	-$40 + $60 = $20 (decide to play)	$13.33
Overall			$0

What we have found is that the Expected Value of Perfect Information was equal to the Expected Opportunity Loss, or: $EVPI = EOL$

Although perfect information is not always unattainable, it’s clear that you’d never want to pay more for imperfect information than for perfect information. This has an interesting consequence: the maximum you should be willing to pay for information is unrelated to the expected value of your default decision. If we changed the game so that you lose $9 million on a 1 or 2, it doesn’t matter, since with perfect information you’ll never risk that loss anyway!

On the other hand, if I occasionally lie to you, you definitely would care if you stood to lose $9 million versus $900. The question is, how much would you care?

Imperfect (misleading) information

In a new and dangerous version of the game, I offer you information for $$k$, but there’s a chance $p$ that I’ll lie to you. This will be represented by $Y$ (for “why would you lie to me?!”).

\[\Pr(y)_Y = \begin{cases} p & lie \\ 1 - p & truth \\ \end{cases}\]

The outcome of the game is determined by both the dice roll $x$, and whether I lied $y$. When I lie, you end up either losing $900, or missing out on $60.

\[H(x, y) = \begin{cases} - k + 0 & x \in \{1, 2\} \cap y = truth \\ - k -900 & x \in \{1, 2\} \cap y = lie \\ - k + 60 & x \in \{3, 4, 5, 6\} \cap y = truth \\ - k + 0 & x \in \{3, 4, 5, 6\} \cap y = lie \\ \end{cases}\]

What is the maximum you should pay $$k$, in terms of the unreliability of information $p$? Solving $E[H(x, y)] \geq 0$ for $k$ and $p$:

\[\begin{align} 0 &\leq E[H(x, y)] \\ &\leq \sum_{x \in \{1..6\},\\y \in \{truth, lie\}} \Pr(x)\Pr(y)H(x, y) \\ &\leq \frac{1}{3}(1-p)(-k) + \frac{1}{3}p(-k - 900) + \frac{2}{3}(1-p)(-k + 60) + \frac{2}{3}p(-k + 0)\\ \end{align}\]

Putting that through the magic math unicorn:

\[k \leq 40 - 340p \\\]

To understand this, consider the extremes:

If I never lie ($p = 0$), I’m giving you perfect information. We’ve already covered that case: $k \leq \$40$, where $40 is the Expected Opportunity Loss of not playing the game.
If I always lie ($p = 1$), then I mislead you every time. I’d have to give you $300 per game to make up for your losses. Note that $300 is the Expected Loss when playing the basic game!

Between $p = 0$ and $1$, the expected value of information varies linearly. If I lie to you less than $p = 40/340 \approx 11.7\%$ of the time, then there exists a price at which it’s worth paying to play.

Let’s take $p=5\%$ and $k=0$ as an example - so I give you the information for free but lie to you 1 in 20 times - and then compute the Expected Value of the game:

Dice Roll	I tell you…	Probability	Value	Expected Value
1, 2	Lose (truth)	1/3 * 19/20 = 19/60	$0 (decided not to play)	$0
1, 2	Win (lie)	1/3 * 1/20 = 1/60	-$900 (mislead into playing)	-$15
3, 4, 5, 6	Win (truth)	2/3 * 19/20 = 38/60	$60 (decided to play)	$38
3, 4, 5, 6	Lose (lie)	2/3 * 1/20 = 2/60	$0 (mislead into not playing)	$0
Overall			-	$23

The expected value of the game - $23 - is exactly what the math above gave us for the maximum we’d want to pay for information: $k = 40 - 340/20 = $23$.

There is another interesting way of looking at this: the Expected Opportunity Loss of the game with 95% truthful information is:

$15 - if you are mislead into playing on a 1 or 2
plus $2 - if you are mislead into not playing on a 3, 4, 5, 6
equals $17.

With the original strategy (don’t play), you stand to lose out on $40 if you’re wrong. With the new strategy (play if I tell you you’ll win), you only risk losing or missing out on $17. Thus, Expected Loss has been reduced by $40 - $17, which is $23!

And that is how How To Measure Anything frames the Expected Value of Information: $EVI = EOL_{before} - EOL_{after}$. That’s why the $EVPI = EOL$, because perfect information reduces the chance of a bad decision to zero.

Imperfect (useless) information

Finally, we can also imagine a version where instead of truth or lies, I give you truth or nothing. When you pay for information but receive nothing, you wasted your money and you just have to stick with your default decision to not play. If there’s even a small chance I might give you the truth then there should be some price worth paying, but if I’ll never tell you anything, then the information is clearly worth nothing. Symbolically, $k \leq 40 - 40p$.

This suggests two categories of imperfect information: potentially useless information, and potentially misleading information.

Potentially useless information might influence you to move from your default decision, but only if it was wrong. For example: an office manager of a small company is deciding whether to provide a vegetarian option for lunch. If the number of vegetarians exceeds a threshold they’ll include the option, but since they’ve not heard that anyone is vegetarian they might default to not doing so. In that situation, a survey of the office can only help, but it could be useless if too many vegetarians miss the survey.

Potentially misleading information might influence you to move from your default decision, even if it was right. Taking the example from the opening of product research, perhaps I’m so confident in the product that my default decision is to go ahead with it. If I run a survey of potential customers, and it’s answered by an unrepresentative slice of the population who all hate it, it could persuade me to abandon the project even if it was a good idea.

Rules of thumb for thinking about the value of information

In summary, the situations examined in this post suggest two guiding principles for valuing information.

Firstly, the most you should pay for information is the amount you expect to lose if your default decision is wrong. This is because the most you’ll pay for information is whatever you would pay for perfect information, and perfect information cannot cause you to change to a bad decision.

Secondly, strive to collect information which is not potentially misleading. When the worst case is that information is useless, you don’t have to consider the cost of switching to a bad decision, but when information can mislead you, then you must take into account the cost of being persuaded to make a bad decision.

It’s worth noting that these principles rely on your initial estimates being vaguely correct, or at least honestly representative of how little you know. Producing calibrated estimates is a skill you can learn (another topic covered by Hubbard in How To Measure Anything).

Ultimately, these were just simple examples. In the real world, there will usually be many variables at play, and decisions are often on a continuum (not just yes/no), so I hope to cover more complex situations in a future post. I wrote this post so that I could feel that I truly understand and believe Hubbard’s statements on the value of information - with worked examples to reference - and in that sense it has been a success for me. I hope there was something interesting in here for you, too!

Estimation of normal distribution parameters from percentiles, in Python

Mon, 27 Apr 2020 03:15:00 +0000

Let’s say you’re going to do some business modeling for a friend who’s thinking of opening a flower shop, producing something like this at the end.

If you ask your friend the following question…

Can you give me the scale and shape parameters of the lognormal distribution representing estimated customer spend?

You’re likely to be given a blank stare in response. You’re more likely to find success with this line of questioning:

What do you think a top 95% customer is likely to spend? What about the bottom 5%? Do you think customers are most likely to spend in the middle of that, or closer to the bottom?

Estimation is difficult, but if you’re doing it at all, thinking about it in terms of percentiles tends to be an intuitive approach.

Unfortunately, statistical software doesn’t work this way. Distributions are parametrized in terms of means, standard deviations, shape parameters, and so on. So, how do we go from “90% of customers spend between $10 and $90” to a probability distribution in software?

Find a paper titled Determining distribution parameters from quantiles
Implement it in Python!

This post simply demonstrates some Python code examples for deriving normal and log-normal distributions from percentile estimates.

If you’re interested in understanding the maths behind this and learning about other distributions, I encourage you to check out the paper. You can also read this blog post, which links to this Python code for Gamma and Beta distributions.

Notebook setup

First we just import some libraries and set up some plotting functions.

from scipy.stats import lognorm, norm
import numpy as np
import matplotlib.pyplot as plt

def show_pdf(ax, dist):
    """ Plot the PDF of a distribution to an axis. """
    minx, maxx = dist.ppf(0.0001), dist.ppf(0.9999)
    x = np.linspace(minx, maxx, 1000)
    ax.plot(x, dist.pdf(x))


def show_percentiles(ax, dist, low, high):
    """
    Fill the bottom 'low'-th percentiles and top 'high'-th percentiles
    of the distribution on an axis.
    """
    minx, maxx = dist.ppf(0.0001), dist.ppf(0.9999)
    x1, x2 = dist.ppf([low, high])    
    xs1 = np.linspace(minx, x1, 1000)
    xs2 = np.linspace(x2, maxx, 1000)
    ax.fill_between(xs1, dist.pdf(xs1))
    ax.fill_between(xs2, dist.pdf(xs2))

If we estimate that 5% of customers spend under $10, and 95% spend under $90, then we can write that as

\[\begin{align} P(X < \$10) = 0.05 \\ P(X < \$90) = 0.95 \end{align}\]

Generalizing, we’re interesed in finding the scale and location of distributions parametrized as follows:

\[\begin{align} P(X < x_1) = p_1 \\ P(X < x_2) = p_2 \end{align}\]

Normal Distribution

Here’s the solution for a normal distribution. Here, $\phi{-1}(p)$ is the inverse Cumulative Distribution Function of the standard normal distribution. Again, check out the paper to see the derivation!

\[\begin{align} \textrm{scale} &= \frac{x_2 - x_1}{\phi^{-1}(p_2) - \phi^{-1}(p_1)} \\ \\ \textrm{location} &= \frac{x_1\phi^{-1}(p_2) - x_2\phi^{-1}(p_1)}{\phi^{-1}(p_2) - \phi^{-1}(p_1)} \end{align}\]

Here’s a python function which implements that:

def norm_from_percentiles(x1, p1, x2, p2):
    """ Return a normal distribuion X parametrized by:
    
            P(X < p1) = x1
            P(X < p2) = x2
    """
    p1ppf = norm.ppf(p1)
    p2ppf = norm.ppf(p2)

    location = ((x1 * p2ppf) - (x2 * p1ppf)) / (p2ppf - p1ppf)
    scale = (x2 - x1) / (p2ppf - p1ppf)

    return norm(loc=location, scale=scale)

In the examples below, I’ve produced two normal distributions using percentile estimates.

The show_percentiles function - which fills in the top and bottom percentiles - demonstrates that norm_from_percentiles works as intended. Given only the percentiles, it fills in charts in at the expected x locations.

The second example demonstrates that the percentiles don’t need to be centered. You can parametrize using any two percentiles!

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

dist = norm_from_percentiles(10, 0.05, 90, 0.95)
show_pdf(ax1, dist)
show_percentiles(ax1, dist, 0.05, 0.95)
ax1.set(title="5% below 10, 5% above 90")


dist = norm_from_percentiles(5, 0.05, 10, 0.50)
show_pdf(ax2, dist)
show_percentiles(ax2, dist, 0.05, 0.50)
ax2.set(title="5% below 5, 50% above 10")
plt.suptitle("Normal distributions");

Log-Normal Distribution

The paper stipulates that for a log-normal distribution, you can simply replace $x_1$ and $x_2$ with $\log x_1$ and $\log x_2$:

\[\begin{align} \textrm{scale} &= \frac{\log x_2 - \log x_1}{\phi^{-1}(p_2) - \phi^{-1}(p_1)} \\ \\ \textrm{location} &= \frac{(\log x_1)\phi^{-1}(p_2) - (\log x_2)\phi^{-1}(p_1)}{\phi^{-1}(p_2) - \phi^{-1}(p_1)} \end{align}\]

and as we shall see, that seems to work! However, following Scipy’s documentation on lognorm, we also need to make a final transformation from this parametrization to SciPy’s shape and scale.

A common parametrization for a lognormal random variable Y is in terms of the mean, mu, and standard deviation, sigma, of the unique normally distributed random variable X such that exp(X) = Y. This parametrization corresponds to setting s = sigma and scale = exp(mu).

Here’s the Python code:

def lognorm_from_percentiles(x1, p1, x2, p2):
    """ Return a log-normal distribuion X parametrized by:
    
            P(X < p1) = x1
            P(X < p2) = x2
    """
    x1 = np.log(x1)
    x2 = np.log(x2)
    p1ppf = norm.ppf(p1)
    p2ppf = norm.ppf(p2)
    
    scale = (x2 - x1) / (p2ppf - p1ppf)
    mean = ((x1 * p2ppf) - (x2 * p1ppf)) / (p2ppf - p1ppf)
    
    return lognorm(s=scale, scale=np.exp(mean))

The examples below use the same parameters as the normal distributions above, but look very different (they have a heavy right-skew). This highlights the importance of choosing the correct distribution.

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

dist = lognorm_from_percentiles(10, 0.05, 90, 0.95)
show_pdf(ax1, dist)
show_percentiles(ax1, dist, 0.05, 0.95)
ax1.set(title="5% below 10, 5% above 90")


dist = lognorm_from_percentiles(5, 0.05, 10, 0.50)
show_pdf(ax2, dist)
show_percentiles(ax2, dist, 0.05, 0.50)
ax2.set(title="5% below 5, 50% above 10")
plt.suptitle("Log-Normal distributions");

In conclusion

Sometimes, the best you have is your best guess, and it’s much easier for people to guess things like “10% of people would pay less than X” than it is for people to directly guess the location and scale parameters of a distribution.

All of this has left me wondering: why isn’t percentile parametrization more commonly found in maths packages and statistics libraries? I suppose it’s just not a core functionality, and as we’ve seen it’s pretty easy to implement if you need, but I feel like wider adoption of percentile parametrization could go a long way to making simple statistical modeling more accessible.

One exception is Guesstimate (of the Flower Shop example from earlier), a great tool which allows users to define and combine a wide variety of distributions using percentiles. That being said, you’re limited to 90% confidence intervals and in Guesstimate, whereas here we’ve demonstrated that you can use any percentiles you like.

Ultimately, this all just scratches the surface of a very interesting topic. For example, Quantile Parametrized-Distributions are parametrized with 1st, 10th, 50th and 90th percentiles. Unlike the normal and log-normal distributions which are forever symmetric and right-skewed, the 5 parameters of QPDs produce an appropriately skewed distribution. An implementation in Javascript which generalizes these ideas even further can be found here.

The math is clearly out there, some people are clearly taking interest in this, and now this implimentation is out there too. Next time you need to model something, consider estimatimating some percentiles!

How To Measure Anything: The Information Values for Multiple Variables

Sat, 14 Mar 2020 19:23:37 +0000

The book How To Measure Anything (by Douglas W. Hubbard) teaches a process for quantitative decision making. I highly recommend giving it a read, but here’s the overall approach it suggests:

Define and model a decision. A decision must have a measurable impact, e.g. company profits, which is modeled with a collection of variables. For example, decide to sell a product if $\textrm{Unit Profit} * \textrm{Buyers} - \textrm{Fixed cost} > 0$
Estimate distributions for each variable using existing knowledge. Ideally, this should be done by experts in the area who have undergone calibration training
Compute the value of obtaining more accurate estimates for each variable.
Reduce uncertainty by measuring the highest-value variables, or by decomposing them into finer grained estimates.
Make a decision once uncertainty has been sufficiently reduced.

I believe the cornerstone of this approach is step 3, but how exactly do you compute the value of measuring a variable?

The book describes the theory in detail, with some nice examples… for a model with a single variable. It then acknowledges that most models will consist of many variables, and dedicates a mere two short paragraphs to the multi-variate case!

This blog post fills in the gap with a working example. I define a model, then compute the value of obtaining perfect information for each of its variables using the more accurate method suggested by the book.

The advertising campaign

The model we’ll use is similar to the simple model used in Chapter 6, but with a second variable included.

There is a proposal for an ad campaign which will will cost $5m to run. If the campaign is rejected, no money will be made or spent. If the campaign is accepted, it will run in two market segments, and may succeed or fail in each segment independently.

In this unrealistic example, we know exactly how much we’ll make if we succeed, but our experts have estimated the chance of success in each segment.

	Chance of success	Revenue if successful	Revenue if unsuccessful
Segment A	60%	$45m	$0
Segment B	70%	$6m	$0

We could run surveys to improve our confidence in the outcomes, but which market segment should we do that for, and how much should we spend?

Intuition should rightly tell you that A is worth more to measure than B, but the hope is that if we can demonstrate Hubbard’s technique on this simple example, you can generalize it to situations where the outcome is less clear.

The value of perfect information

The method starts by computing what it calls the “Overall Expected Opportunity Loss” for the model. Here’s the quote from Chapter 6, page 164 of the Third Edition:

Another more elaborate – but more accurate – method starts by running a simulation that shows our expected loss at our current level of uncertainty. I call this the Overall EOL or the Decision EOL. This is simply the average of all outcomes given our current (default) decision assuming we don’t measure any further. […] All of these results averaged together – zeros and non-zero values alike – is the Overall EOL.

Opportunity Loss (OL) is defined as the cost of a bad decision, which could be as much as $5m for this campaign if it completely fails. Expected Opportunity Loss (EOL) is the cost of a bad decision multiplied by its probability.

If the campaign is run, we risk losing money if we don’t make as much as the $5m we paid for it.

A result	B result	Profit	Probability	Opportunity Loss (loss if campaign is accepted)
A succeeds	B succeeds	$46m	42%	$0
A succeeds	B fails	$40m	18%	$0
A fails	B succeeds	$1m	28%	$0
A fails	B fails	-$5m	12%	$5m

\[\begin{eqnarray} \textrm{EOL if Accepted} &=& \sum_{\textrm{all scenarios}}{\textrm{Opportunity Loss} * \textrm{Probability}} \\ &=& $5m * 12\% \\ &=& $600k \end{eqnarray}\]

Ignoring how much we stand to make, running this campaign involves a 12% risk of a $5m loss, so the expected loss is $600k.

We can make a similar calculation for the case where the campaign is not run. Here, the risk is that we lose out out on everything we would have made.¹

A result	B result	Profit	Probability	Opportunity Loss (loss if campaign is rejected)
A succeeds	B succeeds	$0	42%	$46m
A succeeds	B fails	$0	18%	$40m
A fails	B succeeds	$0	28%	$1m
A fails	B fails	$0	12%	$0

\[\begin{eqnarray} \textrm{EOL if Rejected} &=& \sum_{\textrm{all scenarios}}{\textrm{Opportunity Loss} * \textrm{Probability}} \\ &=& $46m * 42\% \\ &+& $40m * 18\% \\ &+& $1m * 28\% \\ &=& $26.8m \end{eqnarray}\]

Based on what we know, should we decide to run the campaign? That depends on our risk tolerance - if we don’t have $5m to lose, we might be more sensitive to actual losses than missed profits.

However, let’s keep things simple and say that we’ll take whichever option has the lowest risk. This policy is our decision rule. It results in a “default” decision to run the campaign, and means that our Overall EOL is currently $600k.

Just to be safe, should we spend a bit on a survey to refine our estimate of the chance of failure?

Let’s imagine we can buy perfect information. We’d definitely do it for $1, and definitely not for $100m, so what’s the maximum we should pay? The theory goes that given our current level of uncertainty, we would pay up to $600k to confirm that the campaign won’t fail. Despite the fact that we might lose up to $5m, any more than $600k and we’re paying more than our expected loss.

Conversely, if our default option was to reject the campaign, we’re throwing away an ‘expected’ $26.8m, so it would probably be worth spending a lot more to make sure we’re not missing out. However, if we pay more than $26.8m, we’re paying more than we expect to gain.

This is what the book calls the Expected Value of Perfect Information (EVPI), and as you can see, the value of information partly depends on the action you’d take without it. We’ve calculated it for the entire model, but what if we can get perfect information for just one variable?

The actual value of information for a single variable

We then run a series of Monte Carlo Simulations where we pretend we knew one selected variable in the model exactly.

Let’s imagine that we get some new information, and now we’re 100% confident the campaign will fail in segment A.

A result	B result	Profit	Probability	Opportunity Loss (Accepted)	Opportunity Loss (Rejected)
A succeeds	B succeeds	$46m	0%	$0	$46m
A succeeds	B fails	$40m	0%	$0	$40m
A fails	B succeeds	$1m	70%	$0	$1m
A fails	B fails	-$5m	30%	$5m	$0

\[\begin{eqnarray} \textrm{EOL if Accepted} &=& $5m &*& 30\% &=& $1.5m \\ \textrm{EOL if Rejected} &=& $1m &*& 70\% &=& $700k \end{eqnarray}\]

Note that we computed these numbers analytically, but 100 random simulations (i.e. Monte Carlo) would have produced roughly the same results.

We apply a decision rule that tells the model what I would have done differently if I knew only that variable exactly. If knowing this variable was informative, my opportunity losses from making a bad decision (the Overall EOL) should go down.

This is the part of the book I’ve been having trouble interpreting. The new information was clearly informative, since my decision rule now tells me to reject the proposal. However, the expected opportunity loss of my decision has gone up from $600k to $700k! So what was the value of this information?

Was it $0, since my expected losses didn’t decrease?
Was it $100k, the absolute difference between Overall EOL before and after?
Was it $900k, since the expected loss of my previous decision turned out to be $900k larger than I thought?
Or was it $800k, since the new information updates the EOL for my previous decision to $1.5m and then updates my decision to reduce my EOL to $700k?

I argue the the answer is (4), and I’m going to back that up with some extreme examples.

If I had somehow learned that running the campaign risked a billion dollars, with a $50m potential upside, that information would clearly have immense value. I was going to run that campaign before! If I had paid a $100m dollars to avoid an expected billion dollar loss, that would have been very much worth it.
However, if I’d have learned that the campaign stands to make OR lose one one billion dollars – and I value actual loss as much as I value lost potential gains – I would regret having spent any money on research at all. If I have 1000 opportunities to run this campaign, it won’t matter whether I do or not - I’ll make an average of $0. Spending money to learn that would have just cost me extra.

With that in mind, my interpretation if the

Putting that aside, lets try the same thing for the case that we learn that segment A is 100% likely to succeed. Noting that this reduces the situation to a single variable, we can simplify the table a bit.

B result	Profit	Probability	Opportunity Loss (Accepted)	Opportunity Loss (Rejected)
B succeeds	$46m	70%	$0	$46m
B fails	$40m	30%	$0	$40m

\[\begin{eqnarray} \textrm{EOL if Accepted} &=& $0 \\ \textrm{EOL if Rejected} &=& $46m * 90\% + $40m * 10\% = $44.2m \end{eqnarray}\]

Again, perfect information reduced our risk to $0, and our decision rule tells us to definitely run the campaign, because segment A’s success will always cover the potential loss of segment B. Overall, it looks like the expected value of perfect information on segment A is $600k.

Now let’s consider perfect information about segment B. Suppose we learn that it will fail.

A result	Profit	Probability	Opportunity Loss (Accepted)	Opportunity Loss (Rejected)
A succeeds	$40m	60%	$0	$40m
A fails	-$5m	40%	$5m	$0

\[\begin{eqnarray} \textrm{EOL if Accepted} &=& $5m &*& 40\% &=& $2m \\ \textrm{EOL if Rejected} &=& $40m &*& 60\% &=& $24m \end{eqnarray}\]

The campaign looks $720,000 more risky, but that doesn’t change our decision to run it.

What if we learn that it succeeds?

A result	Profit	Probability	Opportunity Loss (Accepted)	Opportunity Loss (Rejected)
A succeeds	$46m	60%	$0	$46m
A fails	$1m	40%	$0	$0

\[\begin{eqnarray} \textrm{EOL if Accepted} &=& $0 \\ \textrm{EOL if Rejected} &=& $46m &*& 60\% &=& $27.6m \end{eqnarray}\]

Now the Expected Opportunity Loss of our decision to run the campaign is reduced by $80,000.

The change in Overall EOL by eliminating uncertainty for a single variable is the “Individual EVPI” for that variable.

Here is where my interpretation of the book is fuzzy. It seems clear that we should pay up to $600k for perfect information on Segment A, but since information on Segment B can never change our decision, would we pay even one dollar to learn more about it?

Knowing what to measure

WIP

this was a simple example
it still demonstrates that P(success of B) is the most valuable estimate to refine
when a model consists of a sum of variables, the variable with the widest range is always going to be the most informative
but in more complex models:
- it won’t always be a sum of variables
- it may be that no single variable is informative

Decision rules, loss functions

WIP

“Decision rule” selects from decisions based on their EOL
Could just take the decision with the lowest EOL, but not always
You might be more sensitive to actual loss than to lost potential gain.
e.g. might have a cap on maximum actual loss (a budget), but not a cap on lost potential gain.

Realistic decisions

WIP

Application to ranges of variables and decisions would follow the same principle
- Hold a variable (or group of variables) at a single value
- Simulate the others & compute EOLs of resulting decisions.
- Compute EOL for

Interlude: does any of this actually make sense?

WIP - worth including?

Linear loss versus maximum loss

The concept of paying [WIP finish this sentence], and ultimately it’s a pretty simple model of risk. Perhaps we think that we should spend up to $5m because there’s some chance we could lose that much. That would be the Maximum Opportunity Loss, and perhaps it sounds like a good idea, but it falls apart for slimmer probabilities. For example, if we introduce a 0.001% chance of a massive PR disaster which costs $500m in lost sales, should we spend $500m to test the advertisement for PR issues? Probably not! The linear model (for this single variable) suggests $5,000 which seems much more reasonable.

Uncalibrated estimates

TODO something about how the model falls apart if you begin with uncalibrated estimates.

######################## TODO ####################

Simulate two agents making a decision

D = p1 * a + p2 * b > c

Agent 1 always chooses the same thing.

Agent 2 learns the true value of p1 or p2, then chooses appropriately

When you realize that loss can including losing out, I think that it makes the name Expected Opportunity Loss more clear. ↩

Simple Custom Resources for the AWS CDK

Sun, 28 Jul 2019 19:23:37 +0000

If you are using the AWS CDK you will quickly discover that because it’s just a wrapper around CloudFormation, it shares with it the issue that not all of AWS’s features have been implemented.

Perhaps you need to enable Aurora’s Data API, which was the case for me. Whatever it is, you know you just need to make a single AWS API call. What’s the simplest thing for you to do?

The not simple approach

Your story might go something like this:

You do some googling and learn from StackOverflow that you can do this with “Custom Resources”, which the CDK supports with CustomResource
You find the cfn-response module which is supposed to abstract away the details of sending a response to CloudFormation to let it know the status of your resource after CloudFormation requests that is is created, updated or deleted.
You try numerous times to actually import it correctly in your Lambda function, which is difficult because:
- The docs ask you use ZipFile but in the CDK you do that using lambda.Code.inline, not by specifying a directory which will get zipped, which is not terribly obvious,
- Every time you make a mistake, CloudFormation gets stuck rolling back because your custom resource is unable to respond to say that it has been successfully deleted,
- You eventually discover that the cfn-response library doesn’t work with a lambda function which uses async anyway, and you don’t want to give up on async because it’s ~the future~.
So you switch to using cfn-response-promise. Now you have to go back to lambda.Code.asset because you’re installing a 3rd party library. Also, remember to return the result of send now or you’ll still get stuck!
You learn the hard way that CustomResource parameter names are title-cased when they’re passed to your lambda function.
You discover that you can’t pass a reference to a CfnDBSubnetGroup via Custom REsource parameters using .dbClusterIdentifier, but .ref seems to work.
When you get through all of this it dawns on you that you still have to give your Lambda function permissions to do whatever it needs to do.

At this point you start to wonder whether there was a simpler way, poke around in the CDK docs and notice something sitting at the bottom… custom-resources.

This library provides a Construct called AwsCustomResource which lets you specify a single SDK call which will happen when my resource is created, updated or deleted. Great! You throw something together:

new AwsCustomResource(this, "ToggleDataAPI", {
  onCreate: {
    service: "RDS",
    action: "ModifyDBCluster",
    parameters: {
    DBClusterIdentifier: db.dbClusterIdentifier,
    EnableHttpEndpoint: true,
    physicalResourceId: `DbDataApiToggle`
  }
});

It looks like cdk synth is happy so you run cdk deploy.

Failed to create resource. awsService[call.action] is not a function
    new CustomResource (.../node_modules/@aws-cdk/aws-cloudformation/lib/custom-resource.ts:92:21)
    \_ new AwsCustomResource (.../node_modules/@aws-cdk/custom-resources/lib/aws-custom-resource.ts:159:27)

Why is this happening? ModifyDBCluster perfectly matches the action name given in the AWS Docs. Perhaps the source code will be enlightening. You find the reference to awsService in the source code for the Lambda function itself.

const awsService = new (AWS as any)[call.service](call.apiVersion && { apiVersion: call.apiVersion });

try {
    const response = await awsService[call.action](call.parameters && fixBooleans(call.parameters)).promise();
    // ...

This code is trying to look up the function you specified in the aws-sdk library but ModifyDBCluster doesn’t exist in AWS.RDS. So what does exist in RDS? Yep, modifyDBCluster with a small m.

Finally you cdk deploy and it works. You took 5 hours to successfully automate a single button press.

Was it worth it?

A better way

The AwsCustomResource class seems like it’s almost a winner when all you need to do is make AWS API calls, but I did not go through all of the above described effort just to live in a world where I discover configuration typos at deploy time, so I made a library called checked-aws-custom-resource which consists of a single resource which just wraps AwsCustomResource and checks at compile time whether all specified actions exist. It’s really pretty simple, but now I feel like I’ll be able to use the CDK to poke AWS APIs without fear.

It’s basically a drop-in replacement:

import { CheckedAwsCustomResource } from "checked-aws-custom-resource";

//...

new CheckedAwsCustomResource(this, "ToggleDataAPI", {
  onCreate: {
    service: "RDS",
    action: "ModifyDBCluster",
    parameters: {
    DBClusterIdentifier: db.dbClusterIdentifier,
    EnableHttpEndpoint: true,
    physicalResourceId: `DbDataApiToggle`
  }
});

its implementation is little more than to subclass AwsCustomResource and add this to the constructor:

if (typeof awsService[call.action] !== "function") {
  throw new Error(`${call.action} is not a function in AWS.${call.service}`);
}

and you can install it yourself with:

npm install checked-aws-custom-resource --save

Thoughts

Overall, I have come away from this experience with a few thoughts:

CloudFormation (and AWS as a whole, really) is a minefield of gotchas,
If there are some simply documented basic best practices for CDK/CloudFormation/Infra-As-Code development out there, I would love to know more,
The CDK smooths over many of CloudFormation’s edges, but also introduces a few.

To be fair to the CDK, it’s mostly been a joy to use, and it’s still experimental, but ultimately, every new layer of complexity creates an opportunity for bugs to hide and abstractions to leak.

For example, take this snippet from the source code of custom-resources:

/**
 * Transform SDK service/action to IAM action using metadata from aws-sdk module.
 * Example: CloudWatchLogs with putRetentionPolicy => logs:PutRetentionPolicy
 *
 * TODO: is this mapping correct for all services?
 */
function awsSdkToIamAction(service: string, action: string): string {
  const srv = service.toLowerCase();
  const iamService = awsSdkMetadata[srv].prefix || srv;
  const iamAction = action.charAt(0).toUpperCase() + action.slice(1);
  return `${iamService}:${iamAction}`;
}

I don’t know TODO, you tell me!

Debugging HTTPS configuration for Dokku

Sat, 06 Jul 2019 19:11:00 +0000

I recently decided to set up my website to use HTTPS. Configuring Github Pages - which hosts this blog - was easy . I just had to follow the Github Pages documentation on troubleshooting custom domains which directed me to update the A records in veryjoe.com’s domain configuration, then check a checkbox.

Dokku - which hosts my side project web apps - should have been just as easy. There is Dokku plugin called dokku-letsencrypt which lets you automatically register and configure Let’s Encrypt SSL certificates. It promises to get you set up with just 3 commands, but I ran into two issues.

It fails if you’ve misconfigured your app’s domain in Dokku.
It fails if your apps have no network listeners.

I’m not going to dig into the precise causes of these errors in this post, since I didn’t do that in real life. But in the unlikely event that you are currently dealing with the same issues, hopefully this helps you debug.

1. Dokku domain misconfiguration

The first site I tried to configure was js.apps.veryjoe.com. Running dokku letsencrypt js produced following error:

ACME server returned an error: urn:acme:error:malformed :: The request message was malformed :: Error creating new authz :: Name does not end in a public suffix

This suggested something was up with Dokku’s Domain name configuration. I investigated with dokku domains:report:

=====> js domains information
       Domains app enabled:           true
       Domains app vhosts:            js.apps
       Domains global enabled:        true
       Domains global vhosts:         apps

root@apps:~# dokku domains:add-global

Apparently I had misconfigured Dokku, supplying it with just apps as the domain name instead of apps.veryjoe.com, so the plugin was trying to request a certificate for js.apps, and Let’s Encrypt wasn’t having it.

I tried to reset the global domain configuration, but that didn’t actually change the configuration for the js app.

root@apps:~# dokku domains:remove-global apps.veryjoe.com
-----> Removed apps
root@apps:~# dokku domains:add-global apps.veryjoe.com
-----> Added apps.veryjoe.com
root@apps:~# dokku domains:report
=====> js domains information
       Domains app enabled:           true
       Domains app vhosts:            js.apps
       Domains global enabled:        true
       Domains global vhosts:         apps.veryjoe.com

I figured that perhaps I just had to clear the app-specific configuration too, and I was right.

root@apps:~# dokku domains:clear js
root@apps:~# dokku domains:report
=====> js domains information
    Domains app enabled:           true
    Domains app vhosts:            js.apps.veryjoe.com
    Domains global enabled:        true
    Domains global vhosts:         apps.veryjoe.com

root@apps:~# dokku letsencrypt js
=====> Let's Encrypt js
...
-----> Certificate retrieved successfully.

In conclusion, Name does not end in a public suffix is pretty self explanatory. You just need to get your domain names in order!

2. missing network listeners

Next I tried diff.apps.veryjoe.com. The error follows (emphasis added):

Unable to reach http://diff.apps.veryjoe.com/.well-known/acme-challenge/zWbBBKkpaQD-6O0NXO8FWktijAIKSpDMWI-2K9MlrVw: HTTPSConnectionPool(host=’diff.apps.veryjoe.com’, port=443): Max retries exceeded with url: /.well-known/acme-challenge/zWbBBKkpaQD-6O0NXO8FWktijAIKSpDMWI-2K9MlrVw (Caused by SSLError(CertificateError(“hostname ‘diff.apps.veryjoe.com’ doesn’t match ‘js.apps.veryjoe.com’“,),))

diff.apps.veryjoe.com was not successfully self-verified. CA is likely to fail as well!

Generating new certificate private key CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or –default_root? Is there a warning log entry about unsuccessful self-verification? Are all your domains accessible from the internet? Failing authorizations: https://acme-staging.api.letsencrypt.org/acme/authz/rx8o_z-L66gQgOVHjWWUoLDlIb9vOEPzTBqPvxnYkEM

Let’s Encrypt tries to access a file which the dokku-letsencrypt plugin hosts in order to prove that I own the domain, but it can’t find it.

Debugging the network error

First I tried looking up “dokku letsencrypt hostname doesn’t match” and found this issue which recommended following some instructions in the plugin’s readme. They didn’t work but I noticed something suspicious while I was following them:

root@apps:~# dokku proxy:ports-add diff http:80:5555
!     No web listeners specified for diff

No web listeners specified suggests some kind of network configuration error. I googled around and found a github issue and a tutorial which connected nginx to issues with dokku-letsencrypt. So why was diff broken in this way but not js?

Eventually I ran dokku network:report:

root@apps:~# dokku network:report
=====> alcoholculator network information
    Network bind all interfaces:   false
    Network listeners:
=====> diff network information
    Network bind all interfaces:   false
    Network listeners:             
=====> js network information
    Network listeners:             172.17.0.4:5000
    Network bind all interfaces:   false
=====> paint network information
    Network listeners:             172.17.0.2:5000
    Network bind all interfaces:   false
=====> thumbnailer network information
    Network listeners:             172.17.0.3:5000
    Network bind all interfaces:   false

Now I had a hypothesis: alcoholculator should also break, and paint should succeed.

Unfortunately, I hit a snag at this point.

Avoiding Let’s Encrypt rate limits

Let’s Encrypt only lets you try to register certificates a few times every three hours. Running dokku letsencrypt paint:

There were too many requests of a given type :: Error creating new registration :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/

The solution suggested by the link is to use the staging environment while you experiment.

dokku config:set --no-restart paint DOKKU_LETSENCRYPT_SERVER=staging
dokku config:set --no-restart alcoholculator DOKKU_LETSENCRYPT_SERVER=staging

Solving the issue

At this point I was able to prove my hypothesis. Running against their staging environment, paint successfully registered for a certificate, and alcoholculator encountered the same error as diff.

The diff and alcoholculator apps are both static Javascript sites, so I figured that the root cause could lie in their buildpack. I inspected their repositories and in both I found a file called .env containing:

export BUILDPACK_URL=https://github.com/florianheinemann/buildpack-nginx.git

As I began to search for issues related to Dokku’s static site buildpack and letsencrypt, I realized that that the latest version of the buildpack actually lives at https://github.com/dokku/buildpack-nginx now. The instructions there tell you to just put a file called .static in the root of your repository. I did that for diff and pushed the changes. Now it had a network listener:

root@apps:~# dokku network:report diff
=====> diff network information
    Network bind all interfaces:   false
    Network listeners:             172.17.0.6:5000

And dokku-letsencrypt diff ran successfully against the staging environment:

root@apps:~# dokku letsencrypt diff
-----> Certificate retrieved successfully.

I decided that it wasn’t worth digging into why exactly the old version was breaking things. Sometimes things break, you update them, and then they’re fixed 🤷‍♂️.

After fixing alcoholculator in the same way, I just had to wait 3 hours for Let’s Encrypt’s rate limits to reset and then I was able to set everything up.

Conclusion

Ultimately both of the problems here were of my own making: misconfiguration and outdated dependencies. The dokku-letsencrypt plugin is pretty great (this tutorial suggests that a Let’s Encrypt certificate would be significantly more painful to set up without it!) but no software can completely prevent user error. With decent error messages and warnings, I was able to web-search my way to solutions. I hope you enjoy your secure connection to my side projects!

I used the formula for the volume of a cylinder in real life for the first time ever and it went OK

Sun, 26 May 2019 19:23:37 +0000

For me, the phrase “pie dee over two squared aitch” (𝜋 (d½)² h) - also known as “pie are squared aitch” (𝜋r²h) - evokes images of university lectures; magnetic cylindrical rods, electric charge in a piece of wire. If you need to understand those things, you need to know how much space they take up in the world. Until the situation described here, I don’t think I’ve ever actually needed to calculate the volume of a cylinder outside of an exam.

Recently I found myself making a pretty large quantity of apple butter. I can’t say I’m apple butter’s greatest fan, but I’ve started cooking in bulk according to precise recipes and storing it all in glass jars at room temperature a.k.a. “canning”, and apple butter is a good introductory recipe.

The recipe looked simple enough, it’s basically apple sauce cooked for longer. The recipe asks you to boil 6 lbs of apples in 2 cups of apple cider for half an hour, blend it, then add 3 cups of sugar, some cinnamon and keep cooking for a few hours.

However, I hit a snag at the “blend it” stage. The recipe book suggested I transfer boiled apples to a blender for puréeing, then transfer them back to the pot. Such unnecessary effort! I’m pretty sure that immersion blenders had been invented by 2006 when this book was written.

I patted myself on the back as I transformed the contents of my pot of apples to mush and saved a significant quantity of cooking and washing labor. This was a figurative patting on the back. I recommend a two-handed grip for stick-blending large quantities of scalding hot apples.

Then I read on. The next step was “Measure 12 cups (3 L) of apple purée”, then the step after was to add the sugar, but if I had too much apple purée the ratio of sugar to purée might be off and so there might not be enough sugar to preserve the final product! I needed to be sure in that moment that I had the correct volume of apples. I resigned myself to having to wash a measuring jug after all

Then inspiration came to me. I ran to grab a tape measure. First I measured the height of the pot, starting from the top of the base.

The height of the pot.

then I measured the distance between the top of the purée and the top of the pot,

The distance between the top of the purée and the top of the pot.

and finally I measured the diameter of the pot.

The pot's diameter.

The two vertical measurements gave me 23 - 17 = 6cm for the depth of the purée, so I plugged in all the numbers. (25cm/2)² ⨉ 𝜋 ⨉ (23 - 17) cm = 2945cm³! Apparently I had managed to produce an almost perfectly correct volume of purée. Happy with this outcome, I followed the recipe to the end.

A batch of apple butter.

However, in retrospect I think I was being a bit cavalier with my measurements, rounding to the nearest 0.5 cm. If I were 0.25cm off the real measurements, how wrong could I be?

If my measurements overestimated the external height and diameter and underestimated the internal height all by 0.25cm, the real volume would have been (24.75 cm / 2)² ⨉ 𝜋 ⨉ (22.75 - 17.25) cm = 2646cm³, and I would have been 12% off the recipe. In the other direction, if the real volume were (26cm/2)² ⨉ 𝜋 ⨉ (23.25 - 16.75) cm = 3505cm³ I would have been 16% over!

That doesn’t seem so bad, but there was another mistake I could have made: if the tape measure was not perfectly vertical then I may actually have measured the longest side of a triangle formed by the side of the pan, the 1.5cm metal tip of the tape measure, and the tape itself, which would have resulted in an overestimate.

Thankfully, the Zhoubi Suanjing taught us that a² + b² = c². In the worst case, if my 23cm outside pan measurement had been angled and overestimated by 0.25cm, the real height would actually be a = sqrt(22.75² - 1.5²) = 22.70 cm, and the loss of 0.05cm would bring the real volume from 2646 cm³ to 2622 cm³. Think about that 40cm³ difference as 20 little-finger-tip sized cubes, or roughly 1/6 of a 8z jar of apple butter.

In any case, later on I messed up by undercooking the apple butter by about 3 hours, noticing my mistake after canning about half of the mixture into four 8oz jars, then cooking the rest for longer and canning it in five more 8oz jars, bringing me to 9 jars instead of the recipe-suggested 8, so in the end it was hard to know how accurate I really was.

But hey, at least I had fun measuring things. If you actually managed to get this far: thanks for your time!

Hello Jekyll

Tue, 27 Sep 2016 04:23:37 +0000

DigitalOcean’s lovely WordPress instance decided it would constantly throw me Database Errors which I couldn’t be bothered to fix because, in the end, why do I need a database at all?

So I’m trying out Jekyll!

# Also:
def apparently_it_has(nice):
  print("code highlighting %s" % "Built in!")

So far I’m pretty happy with it, although it took a little effort to set up. My primary stumbling block was themes not working, but it turns out that if you have

url: http://mysiteblahblah.com

set in your _config.yml file, Jekyll uses that as the absolute URL from which to try to include all static assets (like themes) - not so useful while you’re developing locally!

In any case, it’s nice to be able to just write in markdown and HTML and instantly preview in my browser, and it’s nice to be able to easily and fully manage my own content. I’m hosting this on github pages, so all I have to do is git push origin master and lo, my blog is updated.

California: impressions, words, photos, food

Wed, 19 Aug 2015 04:23:37 +0000

People keep asking: what’s California like? Everyone has funny accents and rain doesn’t exist.

I guess you want more than that. OK, I’ll give you my very first impression: everything is made of grids.

A plane's eye view of American grid layouts

This is a fairly well known fact amongst us Brits. U.S. cities were founded when we knew how big cities would get, so they just laid everything out in grids. Want to give someone an exact location in a city? Just say “On <x street> & <y street>” and they’ll turn up at the intersection of x and y. Neat!

Another funny thing about grids is that I apparently say it like “grits”, which resulted in an Uber driver who was completely baffled that I would say “Everything is made of grits here!”. Why is that baffling?

Grits.

Because this is grits, “a food made by boiling ground maize (also known as corn), and usually served with other flavourings as a breakfast dish, usually savoury.” Basically corn porridge, except they don’t really have the word “porridge” here.

Going back to the bit where people misunderstand me, why does everything think I’m saying “Jeff” when I say “Joe”? It’s happened so much, I think I might rename myself to Jeff.

But anyway, things get weirder.

I have no words.

This, my English friends, is the Chicken and Waffles which the grits came with. Yes, chicken and waffles. Us British folks have a great appetite for American food culture; we have McDonalds and burgers and american style pizza. And there are some things we know that they have but we just reject, such as deep fried butter. And then there’s chicken and waffles.

Why haven’t we heard of chicken and waffles before? Here’s it’s just a Thing. You can just go to a chicken and waffles place. It’s not as common as pizza or burgers, but every decent sized town will have some chicken and waffles. Every English person I have spoken to is baffled by the concept. I asked a Belgian friend if she had heard of chicken and waffles and she said “yes”, then I said, “no, chicken and waffles”. She was disgusted. When my friend Katarina took me to this chicken and waffles joint in Oakland - fairly intoxicated, I will add - I spent the entire time laughing at chicken and waffles, grits/grids, and my own sorry state of intoxication. I feel like I’ve said “chicken and waffles” more times here than any other phrase ever.

But actually, as you’d expect, if you like waffles and also chicken, you’ll like chicken and waffles. Americans took two unrelated food items and slapped them together. Well done! I suppose I could do the same thing. Hmm. Chocolate… blueberry… bagels…

There is no one to blame for this abomination but me.

Chocolate chip bagel with blueberry cream-cheese? Nope OK I’ll leave the food invention to other people. Bagel places here are unsurprisingly Better Than Ours. The best ones have a large varieties of bagels, baked in store, a large selection of cream cheeses, and they let you combine them however you like. Honestly, Izzy’s Brooklyn Bagels</a> is probably the best place to get breakfast on University Avenue, and that’s not just my Jewish heritage talking. But when you go, you should get something more like this:

A sensible, tasty bagel.

I think that was chive cream cheese and sun-dried tomato bagel, or something along those lines. Whatever you get, it’ll probably be better bang for your buck than this place:

The sauce kind of looks like melted plastic.

Why do they serve… boiled/roast potatoes(?) with Eggs Benedict? Anyway, this place on California Avenue made it clear that the U.S. is just as capable as us at doing slightly overpriced breakfast, except they add strange sides that don’t really fit.

On the topic of sides: you know the whole “America has huge portions” thing? Well, it’s often very true, specifically at sit-down restaurants. This was a relatively bourgey (rhymes with Bruges-eee) Mexican joint also on California Avenue: Palo Alto Del Sol, where I had my first ever shot of genuinely tasty Tequila. Apologies for the terrible quality, the iPhone 5S camera isn’t spectacular in low light.

It's like a flag in enchilada form

To give you an idea of the scale of this: they took a normal good-sized plate and then stretched it, and filled it with very dense beans, steak-filled wraps, and rice, and smothered it all in sauce and cheese. I’ve been thinking carefully about this, and I’ve come to the conclusion that finishing all of your food seems more optional in this country than in the UK. They give you too many chips, too much rice, too many potatoes. Sure you can finish it all if you’re hungry, but if you sit there trying to force it all down, people look at you funny. “Why are you trying to finish that? It’s just potatoes!”. Unbridled prosperity can have some interesting consequences.

Anyway, that’s the ‘upper-tier’ Mexican food which the British are mostly used to, but I’m in California, a place which the U.S. bought from Mexico. They have actual Mexican food here. So here’s a question: what is a taco?

Flat things with stuff on them

Many things are tacos, but these are one of the most common kinds! I think I might once have seen tacos like this in the UK, but still they were at some upmarket place in London. But no, my friends, this is also tacos, a blob of meat spooned on top of floppy tortilla bread. You can get one of these babies for $1 from a taco truck - that’s $3-4 dollars for a full meal. Or you can pay three times as much if you want tacos that aren’t as good. Or in this case, you can go to a Hackathon where Pebble rents out a taco truck and gives everyone free tacos. Yay tech! So, what else is Mexican?

It feels odd to put stomach inside your stomach, but #yolo

This is Menudo, basically a bowl of beef stomach, which you would have for breakfast. I was directed by Katarina to rip a corn tortilla up into little pieces and drop them in to “thicken” the soup, which felt kind of exotically unfamiliar. As a whole, I feel like this is the kind of food which many English folk might turn their noses at, but I can tell you that it was delicious. Other tortilla related products: blue tortilla chips!

I try to always qualify "chips" with "tortilla", since if I don't I'll start referring to "crisps" as "chips". That way lies madness.

Yep, blue corn is a real thing, and an unfortunate victim of bad lighting.

In general, food from The Americas in general is more plentiful here. Here’s a messy photo from a mom-and-pop Salvadorian café we went to in San Francisco’s Mission District:

A jumble of half-eaten Stuff on a plate. A note on portion sizes: this was large enough for both of us.

This is papusa, a kind of soft tortilla filled with delicious things like cheese and beans and maybe pork too, served with rice and beans. It’s reminiscent of omelettes or pancakes, in that flat carby/proteiny way. In terms of price:satisfaction ratio, papusa is the best food I’ve had here. It’s fatty and salty and umami but not too oily, every component is delicious, each part complements the other perfectly, there’s nothing pretentious about it. It’s just good, simple food. And it’s from El Salvador, a country which seems to have zero restaurant representation in the United Kingdom. Don’t believe me? I googled “Salvadorian restaurant UK” and all I found was Salvadoran Armada, a “Spanish” restaurant, which has no papusa. A crying shame, if you ask me.

So you’ve seen one of the three food-specialisms in the bay area I’ve identified: food from south of the border. Another one is avocados. Subway has a “local special” with avocados in it. Pebble’s work catering regularly puts avocados in the salad. This is an avocado sandwich with turkey and bacon, the Nifty from the lovely Peninsula Creamery.

The second best thing on the menu, next to Jefe.

Besides its extensive and excellent selection of sandwiches and breakfast items, their menu exhibits something else which seems to be more prevalent in the bay area than elsewhere: a larger selection of cheeses. When you order a sandwich, you’ll then have to decide between one of five different kinds of cheese.

5 different kinds of cheese. Sometimes they make you pick your bread too. It's all too much for me. I just choose randomly these days. Also: "hot links" are sausages in buns.

This experience threw me off at first, since it’s quite unlike the UK where “cheese” basically means “cheddar” unless you qualify it with “cream”. The same place also sells a product called Orange Whip. My English friends, I am about to bring to you a new and amazing recipe that even you at home can make all by yourself. Here are the steps

Blend ice-cream with an orange fizzy-drink (or “soda”)

That’s it. It is beautiful. It tastes like a fizzy orange sweet except that you’re drinking it. Today, however, it occurred to me that I could ask for Orange Whip with an ice-cream that’s not vanilla. Today, I invented the Raspberry Orange Whip.

It just looks like a milkshake, but it tastes like a divine elixir of love

A new-hire was at the counter at the time and I think he wasn’t actually sure how to make it, so he asked his manager. She then called me up to the counter to let me know that “that’s not how we make it here. Are you sure that’s what you want? It might not be good, I just want you to understand that”. She pulled a friendly yet disgusted face which said “you’re not allowed to complain when this is inevitably terrible”. I’m happy to say that she was very wrong. The Raspberry Orange Whip is amazing.

The J1 visa that I’m on is technically a “cultural exchange” visa, with which I am supposed to bring part of the U.S. back home, and also contribute to the U.S. This is my contribution.

Another California Classic is clam chowder, since seafood in general is very popular. So far all I’ve had is a small cup of clam chowder in a fancy hipster bar in downtown Palo Alto (Local Union 271). It tasted like cream of mushroom soup with clams in it, which is good because I happen to like cream of mushroom soup. I will include this heavily adjusted photograph for the sake of completeness.

What is this, a clam chowder for ants? (in fact, the small portion was a welcome change)

I found out about this place from the bartender at the Gravity Wine Bar. I was told that America has excellent service, and I wasn’t sure what to expect. Well, one could dedicate a whole blog post to American service, but instead, here’s a single example.

Tips for tips!

When I idly asked the bartender “do you know any other good spots in the area?” he proceeded to dump his entire knowledge of bars, restaurants and nightlife in this and all neighbouring cities until I heavily hinted that I was very grateful and had more than enough. For service like this, I can give a 25% tip.

But sometimes, America just gets things wrong.

Did they just put stuff on in the wrong order by accident and call it a day?

Why is the tomato on top of the cheese and fillings? This is incorrect. This is not how you pizza. Sure, its delicious and allows you to include more cheese and lets you really taste the tomato sauce, but it’s wrong. And then, biscuits.

Nope.

Anything which we call “cookies”, americans call “biscuits”. The word “biscuit” in America refers to this abomination. It’s a vaguely scone-like ball of bread, which doesn’t really need to exist. Apparently you eat it with gravy, but I declined to put myself through that ordeal out of sheer indignation.

OK, they weren’t actually that bad. But they’re not biscuits. Digestives are biscuits. I argued with Katarina for half a day to convince her that digestives are biscuits. Eventually she agreed on the grounds that digestives are a thing that you dip in tea, and tea is particularly British, so we reserve the right to call them biscuits. However she won’t agree that bourbons are biscuits and not cookies, despite the accompanying cup of tea in Wikipedia’s photo. Americans…

So, surprise! You might have thought you were going to read a varied and interesting analysis of the differences between the Bay Area and the United Kingdom, but in fact I just tricked you into reading a food-blog post with practically every single self-indulgent food snap I’ve taken since landing. Luckily for you, I don’t intend to continue this way. Future planned topics include “socioeconomic observations: urban contrast” and “doorknobs”.

Hummus Pasta Salad

Mon, 08 Dec 2014 04:00:00 +0000

I made this recently by searching for “hummus pasta salad” on google, taking inspiration for a recipe or two and then improvising with what I had.

It’s surprisingly tasty! The flavours of each component stay relatively separated, so the better your olives/feta/tomatoes/hummus, the better your dish will be.

appreciate my blurry tupperware food-photographpy

Ingredients

About 250g of pasta (e.g. Fusilli. Wholewheat is nice/optional)
Three/four tablespoons of hummus. I make my own with this recipe.
A bowl of cherry tomatoes
Half a jar of olives (about 90g?)
A single red pepper
About 200g of feta, chopped (vegan option: omit!)
A glove of garlic, chopped
Spices (e.g. black pepper, paprika, basil)
A bit of oil for frying

Instructions

Add red pepper, half of the tomatoes and garlic to a frying pan, fry on a medium heat for a few minutes, then turn down to low and add the olives and spices.
Meanwhile, mix the hummus with water until it takes on the consistency of cheese sauce. If you didn’t add enough lemon juice before, now’s your chance!
When the tomatoes have just started to fall apart, add the pasta, the hummus and the rest of the tomatoes.
Simmer for a few minutes (more if you made your sauce too thin), adding the feta halfway through. Serve warm or cold!

I hosted CollabPaint! And got annoyed at 4-years-ago-Joe.

Thu, 04 Dec 2014 03:00:00 +0000

I put my old collaborative whiteboard project CollabPaint back up on the internet! Multiple people can log in to a room and paint at the same time, without having to register for an account or sign up.

You can set the size/colour/opacity for your brush and also draw lines, rectangles and ovals. There’s even a chat box and the option to save download the thing you draw.

I wrote CollabPaint about four years ago, and it was one of my first forays in to node.js and socket.io after CollabBox. Like all of my other projects, I never really polished it. It’s still missing a few things such as an undo button (and some overdue bugfixes…), but otherwise it gets the job done!

On future-proofing Code and following best practice

Sadly I had trouble getting CollabBox to run, because I used old crusty versions of all the libraries and never made a package.json file to record which versions of what things I used, so everything is broken. But, here it is on github anyway.

If I could go back four years, I’d tell myself the following set of related things:

Keep a record of which versions of everything you use, if you can
When you start using a new language or ecosystem, make sure you actually look up the best practices for the things you’re using.
For example, with node.js and npm, actually bother to write package.json files to track all your dependancies!

This kind of stuff should really go without saying, but it strikes me as interesting that - as a hobbyist coder four+ years ago - it didn’t really occur to me. So if you ever see your hobby-programmer kids writing code and they forget to make a package.json file, set them straight so they don’t make the same mistakes I did!