The Bias-Variance Decomposition of Human Inequality

Suppose we have two groups of people experiencing different treatment–there is inequality of benefit between the two groups. We can write the difference between their benefits as:


Usually though, we can inspect the size of inequality using quadratic difference, or the quadratic inequality:


Take the expected value:


And rearrange to produce:


This formula can be read off easily. The quadratic inequality between group having a experience and group having b experience is composed of


the mean difference in benefit, i.e. the mean-induced inequality between the two groups is a primal component of inequality,


which is the internal inconsistency of group a and group b also increases the inequality, but it could be removed by


which says if treatment a and b we’re correlated, then that reduces the variance-induced inequality.

To think through an example: suppose I benefit boys by giving or takimg money from each by drawing a random number from \mathcal{N}(2,1). The benefit I give to girls is to drawn from a separate \mathcal{N}(1,100). Clearly boys on average receive $1 more than girls. But also there is an interesting difference in experience. Boys will more consistently receive money around $2, where as the the girls would have money taken or given to them with much more variability approaching $1 only when averaging a large number of experiences. It’s hard to say what the psychological effect of the larger variance is. For example is it unfair for boys or for girls if girls are drawn from \mathcal{N}(2,100)? In this analysis we can clearly detect that there is inequality between the two groups and that the inequality is induced by variance of the benefit to girls.

There is an intersting side note from this branch of social science that in such a circumstance, the number of boys we must sample to ascertain a highly likely interval of his mean reward is lower than the number of girls we must see in order to ascertain the same likely interval of her mean reward. So if the treatment of minority has high enough variability, then it becomes harder to be certain that there is mean-induced inequality. This decomposition enables us to identify such situations.

This is bit of Machine Learning math is really neat! We can now measure inequality in all aspects of life and understand, beyond an average, what is causing inequality and uncertainty.

Where do you live?

It’s march of 2017. Two weeks ago, America has placed an anti-missile facility on a lot in South Korea. If this posts publishes automatically, it’ll either be because we who live in California parishes from a Chinese intercontinental ballistic missile.  Or it may be because I stopped minding my blog.

In either case the situation seems tense. As a resident of California, as a caring human being, I hope this is all for show and no serious escalation occurred. We are the most populus and close point to China. Well with exception of Detroit. But any missile headed east across South Korea has to be headed our way. I hope they are not launched and exploded here.
Let’s hope we survive this one… 🤞

Software Engineers Should Standardize and Unionize (Rated R)

Prostitutes have organized against unfair treatment by society, according to several news sources 1, 2, 3, that police may be disallowed to have sexual relationship with a prostitute in order to arrest the prostitute in hawaii.

The example given in the article says that a police received a blow job from a prostitute (anything other than penetration), and then promptly arrests her for prostitution. Some how the prostitutes felt that this, among other treatments, were demeaning and wrong, and they organized and changed the law.

Software engineers should unionize and make this otherwise proud and difficult professional a respected one, one at least as respected as prostitution.

In one scenario, the manager performs his managerial duties of keeping a project slow or delayed by strategically introducing incorrect information or even outright lies or even blatantly changing documentation or code during some phases of the project and later altering it again to create incorrectness in some part of the software design or coding.

In another situation, the natural abilities of teammates are kept at bay by various nefarious means so that the “right” person, typically an ardent supporter of the manager who is either lacking the knowledge or experience or who just has flexible professionalism and are willing to put in whatever suites management irrespective of actual technical viability. (Sure they are all viable choices and we can write in X86 assembly too, just as long has his gang member knows it well enough)

There is a sorely missing industry standard for software engineering. There is nothing that stands for good (enough) software. The government cannot even put a “Engineer” stamp on software engineering as it can put on structural or civil engineers. Any foreign third grader could come to the US, slap a Mooc in his resume, and become a senior software engineer with rousing praise from others of similar situation a few years earlier… (same applies to domestic self-starters, but please DO NOT come to my home and put dog shit on my lawn, this is an attempt at a civil discussion of my concerns)

Look, guys, look at the Hawaiian prostitutes! They can see something when it is wrong. They stood up against a part of society that is wrong: those who take advantage of prostitutes by enjoying their service–including police man while trying to arrest them. (Of course the accusation is beyond that. Law enforcement actually are accused of soliciting bribery in the form of sexual service for favorable treatment) This blog does not comment on morality of individual state or nation states law enforcement practices. Instead it states a tautological fact: the law strongly discourages the sale and purchase of sexual service. And that should include law enforcement officers. It is an enforcement of something that society has agreed is not right.

Now, can we agree on things that are not right in software engineering?

I don’t imagine that your typical software engineering project my team does to be more complicated than building sky scrapers(think! What would/can president Trump do??):

  • market analysis and requirements
  • product requirement and specifications
  • product design
  • financing, resourcing labor and materials
  • designing and deploying project management
  • project implementation
  • quality assurance
  • completion
  • launch
  • maintenance
  • transfer of ownership
  • remodeling
  • demolition

Think of it! Buildings fall down at far lower rate than you getting sad whale at twitter, or even the once in a while sad robot from Google, when did you ever see a sad leaning Empire State Building?

Think of your salaries and stock options. Think about the quality of service You perform for your fellow man.

Think of the last SEV you caused.

Now think of that builder putting together that building, and the SEV of his work. (Or also, think of those high speed rails in China or Japan, or think of an airbus jumbo jet… there’s a lot of really awesome engineering around the world)

Now think of the government inspectors, zoning, city planning, the federal regulations, the building codes…

Now think of president trump who made some buildings all work.

Now think of the prostitute who demand social justice after being caught sucking a police officers dick. She wasn’t allowed the dignity of washing her hands until she was booked under camera! Think of the dehumanizing experience.

Now think of the last less-than-stellar line of code that you had to write with your hands under duress under recording and review by your company’s overseers, other than that last SEV.

I don’t know why these things mutually activate in my head. But let’s do a association test, do you really feel closer to one of those other engineers than the prostitute? What’s your MTTF?

But all of it could be “solved” if software engineers just had a little pride in their industry and take a stand against “what ever suites the business, the product manager, the engineering manager” or “agile means don’t think, just build, software things naturally and surely will break”

STOP THAT. It doesn’t have to be this way!

We can build something right–risking their exorbitant salaries and stock options of course. Once software engineers standardizes practice and unionize, we would kiss those big pay checks and ISO’s and RSU’s goodbye.

But we can get a button that says “you are a software engineer”

I know I know, a honest to goodness software engineer wouldn’t need a button to know that… and plus look at all the raving reviews on LinkedIn! Isn’t that button enough for you? (No, not really)

(It does not escape me that once we build a reliable car, the demand for or next model would be lower, so let’s build a car that have parts taking turn breaking right after warranted expire and still take on the lost revenue on maintenance…. darn I guess there are other kinds of engineers out there too…)

But we can do better.

There is a lack of objective software building standards. There is a lack of responsibility towards software engineering, instead there is presently only responsiveness for business or politics. There is no pride in this line of work outside of the number next to that $ (or your CTR, or your QPS, or your LOC… take your pick of business metrics)


Let us take a stand and be a better engineering discipline!

In a world… where there are nebulous words!

It occurs to me to write these down on a special occasion… Initially I was considering an optimization problem in a situation where localized optimization is essentially playing a zero sum game with an opponent who is much more powerful.

In this situation, even though we have big data, and even though we have deep learning, it still remains that there is bigger data and yet more sophistication else where. One of the challenges of the nascent big data and deep learning enabled AI industry is one of problem selection.

There are people who are trying to cure cancer and save lives. And there are people trying to trade stocks, win political campaigns, or engage in armed conflict (not that these are the same things) Their continued admonishments against AI are the people who fear the latter. I would imagine there may be very few who would oppose the prior.

That! That is the underlying restriction to the technology: what it can do for prior cause is practically restricted by what it does for the latter. The same applies to all tehnolgies of course. We’ve had internet, social media, a typical Californian would probably take a few minutes to recognize therebeing anything exceeding unusual about the potential downside of yet another meme…

Also, consider aliens, of the interstellar variety, one should always be mindful of our real  competition. There is likely a far greater intelligence out there. Let us not doubt, and let us certainly not delay the development of our own Big Intelligence as matter in due course of our kind’s progress.

Whence be Villainy (retro)

This blog post is being scheduled for far far in the future of actual events that inspired it. All statements are fictional:

At fraud convention, a pair of specialist come on stage and talks about a phishing scam being perpetrated by a desperate Greek. He tricks a stereo typical American broad into committing mail fraud for him and while he’s at it he gets her to buy some stuff for her on credit cards that unbeknownst to her were stolen. But in out experts hands, the bad Greek actor was captured by Interpol and stolen goods were recovered. The broad was slapped around until she smarts.

A few speakers later, another ring was disclosed. This time Asian crime ring. Huge syndicate got us in the back with a Chinese cleaver. We never saw it coming. Thankfully we have gathered the pattern of attack for future preventative care! Another heroic win, sort of.

Do you hear any problems with this? Maybe not if you are an American. We are used to being gouged or stolen from by those poorer, lesser developed, lesser democratic, not-chosen, further-from-God, faceless, moraless creature swarms. What’s next? We’re ready for Algerian prince, Latin American drug lords, or the polean Penguin?

Quickly commissioner, to the bat signal!

From whence shall our next major villain emerge? Where ever it is, it matters little as we will be ever vigilant and perpetually watching from our cave deeply buried under all that worldly wealth.

You better watch it! Evil doers and evil doer’s henchmen! Because we are watching you!

Gross do not read(retro)

By some random brain crossing I remember a conversation at my son’s first pediatrician visit at 24 hrs. I asked the doc why his sexual organs seems much smaller than when he had first emerged the womb with.

The doc explained that there is a lot of trauma, rubbing, squeezing, etc. during birth that causes the components to swell  as all human tissue due when squeezed Or rubbed excessively.

Okay, so…, this kind of means that his enlargements are due to rubbing on his mother’s vagina?And all natural births of men by women mothers experience the same.

Kinda gross. But it undeniably real.

Democracy is humanism

Been reading this book called Sapients. It is a fascinating overview of many important ideas. I found the title sake of this posting very interesting. But it sounds obvious after hearing it. Of course democracy, that which is based on input by its members, is founded on the believe that it’s members are of prime importance and most intelligent and deserving beings of the world.

Actually, I do wonder if this is all said in one breadth the same way throughout history.

Did man, in throes of desire for freedom and respect and political power, out of self interest, not settled on democratic rule as matter of experimentation and thereby selecting a government that worked, finally?

Some longs time later, did founding fathers of America all shared genuine concern for fellow man and then create the American government and those for which it stand?

Truthfully, today, would an argument of humanism sufficiently motivate us to create the same government? I feel that it is more likely to win the argument that it works (or at least doesn’t not work badly enough) than for people to respect humanity enough for that. It might be just me, but I do not see or feel the humanity in most people I see. At least not enough humanity that I would be willing to create a democracy out of trust and respect for their humanity and intelligence.

In realty, most of our organizations are not democracies. I think if you search your thoughts and count the number of people you trust your rights and valuables to, you will find yourself in lackluster company.

How much do we really believe in democracy? How humanistic is it? How do we understand it? 

I wish someone knows the answer… and may that entity be a human person.

Wikipedia Dependency

I find that I can spend a lot of money on a book on a subject but wiki still makes the subject most clear in far far shorter time.
This is becoming a problem. I am less and less able to read longer expositions. Less patience and probably reduced mental capacity to hold longer strands of thoughts. As a species, wiki-style knowledge transfer improves our knowledge sharing, as a person this drastically reduces my own distinctiveness and competitiveness. I may, in fact, be organizing my thoughts as wiki articles. I can say every thing I know in a few minutes, and they are all incredibly clear and right.
I am frustrated with everything else: people speak in imprecise and unedited ways–I can’t stand it, need to ask for clarificationnof every thing! Books do not have introductory paragraph that actually introduces the ensuing content of discussion–what will I be spending next few hours on? Idk! TV will conveniently cut away when vital information should naturally be revealed–and there should be a infographic explaining the relationship between all the characters!!! I can’t stand not knowing the definition of everything–all of which is only available through a link on wiki.
In the future, where we actually do depend on wiki for knowledge, how should it maintained? Admittedly the current management has done well, but when all of humanity shifts to depending on wiki for up to 50% or even 80% of the facts they depend on, there should probably be more thoughts on how it should be maintained.
Not to be worrying about malicious or political edits that the website can have. And further, not worrying about psychological and evolutionary impacts when everyone has access to high quality information. Not considering the possible problems associated with monopolistic situations such as Wikipedia.
If it becomes a public utility, should it not be regulated as public utility. Granted the foundation is an American incorporated organization it already comes with a lot of American values: non-discrimination, nonprofit, apolitical, etc–it is already regulated.
But that regulation is not sufficient for a public utility that a large proportion of the population depends on in a way vey much like how they depend on roads, electricity, water, the weather report, etc. some guarantee of universality must be made to ensure every human has access to knowledge. Some higher level of backup and guarantee of reliable availability in times of crisis. Stated another way, this is mainly to say that more financial resources and more social procedures to safeguard the utility(usefulness, universality and availability) and righteousness(adherence to American values) of Wikipedia and related internet establishments. I’d love for a portion of my taxes to pay for its upkeep, if there comes a time that government regulation are so strong that it becomes part of the government  operations(e.g. USPS, military, intelligence, education, roads, etc), when that is established.
In the same breath, we should say that human knowledge loves freedom. If there is any person in the world who knows of freedom, and who values freedom, and who insists on freedom, that person is with high likelihood a knowledgeable one. Knowledge will resist restriction to the extent of self-destruction. If we do impose any additional restriction not yet ingrown organically, it may be ruined. 
Must tread carefully.

Must think more on the matter.

New Contest II

Last time we considered my need for constructive contests. Recall that we have  now distinguished contest of an hypothetical nature and those contest with real world benefits beyond the purpose of the game. It was also a concern that some games are not fun.

Recently I mentioned to a coworker that brute-force solutions to games are so 20th century. Deep blue already proved it was possible. A large neural network is so 2010's, and alphaGo didn't just beat one world champion, it beat a lot of champions online as well. 

What about this? Let's have a competition that is played by human and computer together and that their enegy consumption is a divisor in the calculation of final score.

This is along the same line of thought from a post long ago before New Contest considerations: automated race car driving competitions where in addition to automotive restrictions that there will be CPU restrictions. Core limited to less than 2^16 cores, each running at no more than 4GHz, total power consumption is to be less than 1kw, etc.

So in a similar idea, one can imagine there be additional restriction on storage. The code the ANN model, the memory about current course and opponents, these all take memory space. So the contest could have an additional rule that there be a 256Gb limit to onboard memory, of any kind, hdd, ssd, cache, ram, GPU ram, etc. 

Suppose the contestants scored S in a scale where each score is real number originally associated with each player. and the highest score wins, the score used to determine the competition result can be S/C where represent the costs paid by competitors to win. For one example this could be the amount of code executed as measured by storage space or as measured by records kept on the CPU itself.

Therefore, in the post-AI years, we will have new contests: the winners must not only accomplish the objectives, they must also complete them efficiently.