The Bias-Variance Decomposition of Human Inequality

Suppose we have two groups of people experiencing different treatment–there is inequality of benefit between the two groups. We can write the difference between their benefits as:


Usually though, we can inspect the size of inequality using quadratic difference, or the quadratic inequality:


Take the expected value:


And rearrange to produce:


This formula can be read off easily. The quadratic inequality between group having a experience and group having b experience is composed of


the mean difference in benefit, i.e. the mean-induced inequality between the two groups is a primal component of inequality,


which is the internal inconsistency of group a and group b also increases the inequality, but it could be removed by


which says if treatment a and b we’re correlated, then that reduces the variance-induced inequality.

To think through an example: suppose I benefit boys by giving or takimg money from each by drawing a random number from \mathcal{N}(2,1). The benefit I give to girls is to drawn from a separate \mathcal{N}(1,100). Clearly boys on average receive $1 more than girls. But also there is an interesting difference in experience. Boys will more consistently receive money around $2, where as the the girls would have money taken or given to them with much more variability approaching $1 only when averaging a large number of experiences. It’s hard to say what the psychological effect of the larger variance is. For example is it unfair for boys or for girls if girls are drawn from \mathcal{N}(2,100)? In this analysis we can clearly detect that there is inequality between the two groups and that the inequality is induced by variance of the benefit to girls.

There is an intersting side note from this branch of social science that in such a circumstance, the number of boys we must sample to ascertain a highly likely interval of his mean reward is lower than the number of girls we must see in order to ascertain the same likely interval of her mean reward. So if the treatment of minority has high enough variability, then it becomes harder to be certain that there is mean-induced inequality. This decomposition enables us to identify such situations.

This is bit of Machine Learning math is really neat! We can now measure inequality in all aspects of life and understand, beyond an average, what is causing inequality and uncertainty.

Where do you live?

It’s march of 2017. Two weeks ago, America has placed an anti-missile facility on a lot in South Korea. If this posts publishes automatically, it’ll either be because we who live in California parishes from a Chinese intercontinental ballistic missile.  Or it may be because I stopped minding my blog.

In either case the situation seems tense. As a resident of California, as a caring human being, I hope this is all for show and no serious escalation occurred. We are the most populus and close point to China. Well with exception of Detroit. But any missile headed east across South Korea has to be headed our way. I hope they are not launched and exploded here.
Let’s hope we survive this one… 🤞

Software Engineers Should Standardize and Unionize (Rated R)

Prostitutes have organized against unfair treatment by society, according to several news sources 1, 2, 3, that police may be disallowed to have sexual relationship with a prostitute in order to arrest the prostitute in hawaii.

The example given in the article says that a police received a blow job from a prostitute (anything other than penetration), and then promptly arrests her for prostitution. Some how the prostitutes felt that this, among other treatments, were demeaning and wrong, and they organized and changed the law.

Software engineers should unionize and make this otherwise proud and difficult professional a respected one, one at least as respected as prostitution.

In one scenario, the manager performs his managerial duties of keeping a project slow or delayed by strategically introducing incorrect information or even outright lies or even blatantly changing documentation or code during some phases of the project and later altering it again to create incorrectness in some part of the software design or coding.

In another situation, the natural abilities of teammates are kept at bay by various nefarious means so that the “right” person, typically an ardent supporter of the manager who is either lacking the knowledge or experience or who just has flexible professionalism and are willing to put in whatever suites management irrespective of actual technical viability. (Sure they are all viable choices and we can write in X86 assembly too, just as long has his gang member knows it well enough)

There is a sorely missing industry standard for software engineering. There is nothing that stands for good (enough) software. The government cannot even put a “Engineer” stamp on software engineering as it can put on structural or civil engineers. Any foreign third grader could come to the US, slap a Mooc in his resume, and become a senior software engineer with rousing praise from others of similar situation a few years earlier… (same applies to domestic self-starters, but please DO NOT come to my home and put dog shit on my lawn, this is an attempt at a civil discussion of my concerns)

Look, guys, look at the Hawaiian prostitutes! They can see something when it is wrong. They stood up against a part of society that is wrong: those who take advantage of prostitutes by enjoying their service–including police man while trying to arrest them. (Of course the accusation is beyond that. Law enforcement actually are accused of soliciting bribery in the form of sexual service for favorable treatment) This blog does not comment on morality of individual state or nation states law enforcement practices. Instead it states a tautological fact: the law strongly discourages the sale and purchase of sexual service. And that should include law enforcement officers. It is an enforcement of something that society has agreed is not right.

Now, can we agree on things that are not right in software engineering?

I don’t imagine that your typical software engineering project my team does to be more complicated than building sky scrapers(think! What would/can president Trump do??):

  • market analysis and requirements
  • product requirement and specifications
  • product design
  • financing, resourcing labor and materials
  • designing and deploying project management
  • project implementation
  • quality assurance
  • completion
  • launch
  • maintenance
  • transfer of ownership
  • remodeling
  • demolition

Think of it! Buildings fall down at far lower rate than you getting sad whale at twitter, or even the once in a while sad robot from Google, when did you ever see a sad leaning Empire State Building?

Think of your salaries and stock options. Think about the quality of service You perform for your fellow man.

Think of the last SEV you caused.

Now think of that builder putting together that building, and the SEV of his work. (Or also, think of those high speed rails in China or Japan, or think of an airbus jumbo jet… there’s a lot of really awesome engineering around the world)

Now think of the government inspectors, zoning, city planning, the federal regulations, the building codes…

Now think of president trump who made some buildings all work.

Now think of the prostitute who demand social justice after being caught sucking a police officers dick. She wasn’t allowed the dignity of washing her hands until she was booked under camera! Think of the dehumanizing experience.

Now think of the last less-than-stellar line of code that you had to write with your hands under duress under recording and review by your company’s overseers, other than that last SEV.

I don’t know why these things mutually activate in my head. But let’s do a association test, do you really feel closer to one of those other engineers than the prostitute? What’s your MTTF?

But all of it could be “solved” if software engineers just had a little pride in their industry and take a stand against “what ever suites the business, the product manager, the engineering manager” or “agile means don’t think, just build, software things naturally and surely will break”

STOP THAT. It doesn’t have to be this way!

We can build something right–risking their exorbitant salaries and stock options of course. Once software engineers standardizes practice and unionize, we would kiss those big pay checks and ISO’s and RSU’s goodbye.

But we can get a button that says “you are a software engineer”

I know I know, a honest to goodness software engineer wouldn’t need a button to know that… and plus look at all the raving reviews on LinkedIn! Isn’t that button enough for you? (No, not really)

(It does not escape me that once we build a reliable car, the demand for or next model would be lower, so let’s build a car that have parts taking turn breaking right after warranted expire and still take on the lost revenue on maintenance…. darn I guess there are other kinds of engineers out there too…)

But we can do better.

There is a lack of objective software building standards. There is a lack of responsibility towards software engineering, instead there is presently only responsiveness for business or politics. There is no pride in this line of work outside of the number next to that $ (or your CTR, or your QPS, or your LOC… take your pick of business metrics)


Let us take a stand and be a better engineering discipline!

Whence be Villainy (retro)

This blog post is being scheduled for far far in the future of actual events that inspired it. All statements are fictional:

At fraud convention, a pair of specialist come on stage and talks about a phishing scam being perpetrated by a desperate Greek. He tricks a stereo typical American broad into committing mail fraud for him and while he’s at it he gets her to buy some stuff for her on credit cards that unbeknownst to her were stolen. But in out experts hands, the bad Greek actor was captured by Interpol and stolen goods were recovered. The broad was slapped around until she smarts.

A few speakers later, another ring was disclosed. This time Asian crime ring. Huge syndicate got us in the back with a Chinese cleaver. We never saw it coming. Thankfully we have gathered the pattern of attack for future preventative care! Another heroic win, sort of.

Do you hear any problems with this? Maybe not if you are an American. We are used to being gouged or stolen from by those poorer, lesser developed, lesser democratic, not-chosen, further-from-God, faceless, moraless creature swarms. What’s next? We’re ready for Algerian prince, Latin American drug lords, or the polean Penguin?

Quickly commissioner, to the bat signal!

From whence shall our next major villain emerge? Where ever it is, it matters little as we will be ever vigilant and perpetually watching from our cave deeply buried under all that worldly wealth.

You better watch it! Evil doers and evil doer’s henchmen! Because we are watching you!

Gross do not read(retro)

By some random brain crossing I remember a conversation at my son’s first pediatrician visit at 24 hrs. I asked the doc why his sexual organs seems much smaller than when he had first emerged the womb with.

The doc explained that there is a lot of trauma, rubbing, squeezing, etc. during birth that causes the components to swell  as all human tissue due when squeezed Or rubbed excessively.

Okay, so…, this kind of means that his enlargements are due to rubbing on his mother’s vagina?And all natural births of men by women mothers experience the same.

Kinda gross. But it undeniably real.

Democracy is humanism

Been reading this book called Sapients. It is a fascinating overview of many important ideas. I found the title sake of this posting very interesting. But it sounds obvious after hearing it. Of course democracy, that which is based on input by its members, is founded on the believe that it’s members are of prime importance and most intelligent and deserving beings of the world.

Actually, I do wonder if this is all said in one breadth the same way throughout history.

Did man, in throes of desire for freedom and respect and political power, out of self interest, not settled on democratic rule as matter of experimentation and thereby selecting a government that worked, finally?

Some longs time later, did founding fathers of America all shared genuine concern for fellow man and then create the American government and those for which it stand?

Truthfully, today, would an argument of humanism sufficiently motivate us to create the same government? I feel that it is more likely to win the argument that it works (or at least doesn’t not work badly enough) than for people to respect humanity enough for that. It might be just me, but I do not see or feel the humanity in most people I see. At least not enough humanity that I would be willing to create a democracy out of trust and respect for their humanity and intelligence.

In realty, most of our organizations are not democracies. I think if you search your thoughts and count the number of people you trust your rights and valuables to, you will find yourself in lackluster company.

How much do we really believe in democracy? How humanistic is it? How do we understand it? 

I wish someone knows the answer… and may that entity be a human person.

New Contest II

Last time we considered my need for constructive contests. Recall that we have  now distinguished contest of an hypothetical nature and those contest with real world benefits beyond the purpose of the game. It was also a concern that some games are not fun.

Recently I mentioned to a coworker that brute-force solutions to games are so 20th century. Deep blue already proved it was possible. A large neural network is so 2010's, and alphaGo didn't just beat one world champion, it beat a lot of champions online as well. 

What about this? Let's have a competition that is played by human and computer together and that their enegy consumption is a divisor in the calculation of final score.

This is along the same line of thought from a post long ago before New Contest considerations: automated race car driving competitions where in addition to automotive restrictions that there will be CPU restrictions. Core limited to less than 2^16 cores, each running at no more than 4GHz, total power consumption is to be less than 1kw, etc.

So in a similar idea, one can imagine there be additional restriction on storage. The code the ANN model, the memory about current course and opponents, these all take memory space. So the contest could have an additional rule that there be a 256Gb limit to onboard memory, of any kind, hdd, ssd, cache, ram, GPU ram, etc. 

Suppose the contestants scored S in a scale where each score is real number originally associated with each player. and the highest score wins, the score used to determine the competition result can be S/C where represent the costs paid by competitors to win. For one example this could be the amount of code executed as measured by storage space or as measured by records kept on the CPU itself.

Therefore, in the post-AI years, we will have new contests: the winners must not only accomplish the objectives, they must also complete them efficiently.

TAS for Chomsky

Continuing the thoughts around expressing Chomsky’s ethical opinion regarding US military actions as expressed by the documentary that I watched during 2016-2017 break.

It became obvious that while propositional logic and set theory are comforting languages, they are not the most convenient when operationalizing an ethic. For example, there seem to be a need to distinguish these prescriptive targets: must do, must not do, may do, may not do.

\detokenize{must_do_actions}\subseteq \detokenize{may_do_actions}

\detokenize{must_not_do} = \detokenize{may_not_do}

It seems “may not do” is “not may do” due to common English use even though technically both expression should be subset relation. I.e. “You may not smoke” states “you must not smoke” instead of “you can choose not to smoke.” Although that imprecision is inconsequential to the current discussions.

So in fact the expression can be simplified by including do_not_do_ actions for each action in simple TAS. We must also impose a contemporaneous interpretation to a \in TAS as to mean a is an action viewed at some references moment, ostensibly now. 

Then, the predicate must\_do(a, p1, p2), reads “p1 must do a to p2” is expressed as ethical(a, p1, p2) \land \neg ethical(do\_not\_do\_a, p1, p2). may\_do is simply ethical.

may\_not\_do is \neg ethical
The need to restrict preposition to single moments in time is in retrospect necessary. All preposition can be sub-indexed with reference time: prescriptive proposition ethical_t means it is ethical at time t, while descriptive preposition do_t means something is done at time t. do_w(A,ally, axis) \implies ethical_u(A, axis, ally) \forall{w,u} \in WWII

That’s a mouthful. But at least we can avoid the pitfalls such as the “may not” fiasco we have in English.
(Disclaimer, I watched a thirdprty documentary about Chomsky. Some videoed statements were stitched together and I watched that. doesn’t mean I am writing to explain what he actually said or meant.)

The sweet spot between CSS and OSS

I’ve been hacking on a commercial database at work recently. Spent a good week of time querying the database for a good sample of a decade of the company’s historical data. The thing that really kills me is this 15 minute query sniper that has been around since before I joined the company–it exist on all data platforms we have ever had: MySQL, postgresql, hive, vertica, spark, Hadoop.

But in an ironic way, I am actually really glad that all this crap works at all! How recent was it that MySQL would just get stuck or run out of memory or some other hidden unknown problem? How recent was it when hive crashed? How recent was it when you had to query hbase for data? But all that is under control because the company managed to pay for a product that actually works. system-v (anonymize to protect the company) can actually handle the work load that we have doing funky 5-level deep subqueries, multiple mixed inner and outer joins, filters group-by’s, aggregations, string operations, and it never peeped a single complaint! Just runs until query is killed. Production etl was not impacted. Other analysts didn’t complain. All I got was when the query got big, it was killed. 

Of course this two dozen person team, director level senior management and all, and all those servers and licensing fee, and all those training classes, and maybe a few quarters of ramp up is more expensive than the two weeks I spent bringing up a spark cluster on an HP laptop and Dell server. My cluster, btw handled the same sized query fine on spark. I only had to upgrade the disks slightly from factory default.

What this illustrates is that closed source software is catching up with Open source software! This is the sweet spot where closed source is at parity in performance and feature when compared with OSS lacking only cheap install and maintenance. Everything works every where. All you choose is your price and reliability. This is where software should stay for ever! Any geek can code up any new algo in matter of seconds. Test it, launch it in the next release. Beat the CSS by a quarter or two to the market. Companies that chose not to use OSS has to wait the few quarters, but that is a choice that they now have! CSS actually must keep up with OSS to stay afloat. OSS is no longer the only choice for real features, performance and non-stupid implementations.

Competition is so awesome for consumers!