The Deep Universal Regressor

There’s this idea in Deep Learning that Neural Networks are universal function approximates. They can approximate any function you can provide data for.

It has confounded me for a long time exactly how it does this for continuous valued output, but recently, through the grapevines that is the Deep Learning community, I finally discovered one answer to this question.

Consider some deep neural network taking in X and producing some kind of penultimate layer of activations, A. We want to write a formula for producing a Y that approximates \hat{Y}.

Oh boy, who are we kidding, let’s just drop down to tensorflow code…

You want to do

Y = inverse\_sigmoid(tf.mean(tf.sigmoid(A)))

Being careful, of course, to calculate the pooling not for the batch but for each input and not to double sigmoid-activate A, but the last activation must be sigmoid-compatible. Note since sigmoid produces numbers between zero and one inclusive, so the mean, or any convex combination, of a bunch of such numbers can also exactly span that range, suitable for input to the inverse\_sigmoid. And of course if you need to, A could have been activated with the likes of tf.exp or tf.square and then filtered through the tf.sigmoid

For example, if you think \hat{Y} ultimately grows with tf.log(A), and you have already made sure A is positive, then you can use the following by simplifying out the exponential and compute

Y = inverse\_sigmoid(tf.mean(\frac{A}{A+1}))

The sigmoidsigmoid^{-1} pair can also be replaced with other bounded activations like the tanh or \frac{x}{\sqrt[1/k]{1+x^k}} and their respective closed-form inverses.It can also be replaced with unbounded constricting activations such as x^{\frac{1}{2k+1}} and x^{2k+1} pair for a chosen whole numbers k.

Tada!

This solves the problems of your deep neural network needing constricting nonlinearities like the sigmoids, your need to produce a continuous output that may grow at non-linear rates relative to activations, your limited computational resources, and your having a lucid hunch as to how they are related.

Hopefully this helps you and saves significant amount of brain activity and experimentation. Your problem will probably need a special architecture using a configuration of this pooling later.

P.s. the use of sigmoidal functions seems beckons to a probabilistic interpretation. The desigmoid, that’s inverse sigmoid, can be interpreted as a lookup from the CDF of a random variable, the value at which it achieves that accumulation. Essentially, in the most basic configuration, this regressor uses each element of A in the penultimate layer to support(or to reflect evidence that) that the desire Y is larger. In a human brain, this positive-only thinking seem overly restrictive. What if we have a field of A is a positive signal that strictly means smaller Y? One way is to use a second FCL to remove effect of one sigmoid from another. A second intuitive idea would be to do the following:

Y = tf.atanh(tf.mean(tf.tanh(A)))

In one step, this regressor can consider both support for larger and for smaller value of Y.

P.p.s. Want to also put in a plug to our wonderful democracy. The computation of mean is explicitly mixing votes of each activation in the penultimate layer equally–each neuron gets an equal vote as to the result. Politics aside, and in addition to convex combinations, all other range preserving combination are fair game–e.g. geometric mean, softmax,etc. depending on the relationship between X and Y and the network that produced A.

Smile, Pay and Hope

So, this fat old lady cuts in front of me at the Los Altos Whole Foods meat counter…

After a few seconds I say loudly to the clerk “I’m next in line, please do not let anyone else cut in front of me.”

“Oh sorry, we’re you in line? I didn’t see you!” The fat old lady said in pretentious seriousness.

This is one very maddening experience just because there are three Chinese looking people standing in line, and all Chinese people look alike, it doesn’t mean that three separate Chinese people are the same person! We’re all in line and we each get a turn. And yes, you are thinking but she could have legitimately thought you were all in the same.e family. With three separate shopping carts, it would be a very insulting scenario she has in her mind for a family of Chinese to push three shopping carts! What are we like pigs and eat a cart full of food each?!

Some days, it’s kind of important to put on that smile and carry on like the world doesn’t have any gunk like I just see.

Whole foods used to be so friendly… Perhaps this also speaks to Amazon’s ownership?

This past weekend we went to buy some chickens from a local poultry vendor. Wow, that was like a very sad experience. I drove an hour with my daughter to her store . She tries to sell us heat lamp and feed stuff. When I looked down to check her prices versus Amazon’s, I got an earful of anti-competition rant about how Amazon squeezed everyone out. “I’m not going to tell you anything about these chickens if you ask for Amazon’s prices.”

I started to talk capitalists sense into her, to explain price competition is good. Organized production is more prolific… But I thought better of it. I fear various kind of retribution for additional expression of disagreement. I’m this wilderness, maybe her gang of crew can come chasing after me with their ATV’s and pitch forks,… Or guns,… Who knows what could happen.

I put on a smile and paid full price.

It was only a smile and extra money to pay to facilitate the happiness of the people we care about. Hopefully all that politics doesn’t get in the way of our common pursuit of individual happiness.

Perils of having being associated with startups

I worked at a lot of San Francisco Bay Area startups (aka Silicon Valley startups) There is a really serious elitist attitude here that you have to be l33t and hack the everything in order to hack a startup.

What that generally means, at least in one respect, is that you have to be smart enough and quick enough to take opportunistic advantage of your surroundings: the competitors(aka incumbent disruptees), some compound economic conditions, but also your customers and investors as needed.

I mean sure, I love all these people trying to do something truly better for their customers and break the old and decadent to create the new and efficient. I get that. And I’m all for that.

But, take this example of what happened to me when I tried to transfer some stock out of the Robinhood platform. I initiated a partial account transfer out of Robinhood to E*TRADE because Robinhood could not provide some service regarding those shares. A few days later, half way through setting up a bracket, Robinhood deactivates my account. It canceled every single outstanding trade I had in flight. And it also nicely sent an email for each of those actions, piling up a screen full in my mail client.

Some thoughts race through my mind… shoot! Did I trigger some kind of secretive financial services rule for brokerage firms that warranted Robinhood deactivating my account and reporting me to an anti-illegal-activity agency?? Did they think I was laundering money by transferring these shares in and out? Did they think I was trying to evade tax and report me to the IRS??? Wait! No no, those are clandestine snitching and invisible to me. This type of deactivation with no explanation usually has to be for something that immediately impacts the company’s business.

And then it dawned on me, yes it does affect Robinhood business! This startup lives off DAU’s and subscription fees (and for god’s sake I even bought gold membership just to patronize fellow startup) making it prohibitively difficult to leave its services. It is to its business advantage to do this. And customer is the wiser until he tries to get his money or stock out of the system, and by then, as it is for me, all is too late.

I’m not picking on Robinhood, it’s a great free service. And because it is so great it really shouldn’t have snuck in that hack into it. For goodness sake! I did a partial transfer and left a lot of money and shares in Robinhood so I can continue to trade!

Wait, are they picking on me because of my race? I know it’s really easy to detect that nowadays with their app reporting my location and the spelling of my name… are they picking on me because my account is too small compared to other users?

The support email responded after almost 24 hours. It claims that they do this to everyone: https://support.robinhood.com/hc/en-us/articles/115001535326-Stock-Transfer.

What happens to my stocks during the transfer?

While stocks are being transferred, you’ll be restricted from trading them until the transfer is complete. You still own the stocks during this process, and they’ll update to reflect the current market value.

Their public documentation indicates that only shares being transferred cannot be traded not all shares! And while it took two days for me to receive this information, i hade the time to call E*TRADE twice to confirm that the transfer was indeed a partial account transfer (of just one stock among maybe two dozen that I own at Robinhood)

This reminds me of my own work at other companies… sometimes technological choices render the upkeep of documentation and sprawling knowledge base impossible. But wait… no the support sent me the link so it must be current as of now?

And certainly there are cases where an unnoticed bug prevented some metric driving feature from being disabled by customers in many situations… sure everyone hacks here.

But it doesn’t need to do that! It’s such a great service, free !! I mean how do I even ask for remediation of my trading losses from a brokerage firm that doesn’t charge money to trade? Is the truism “you get what you paid for” really true? And yet I feel outraged. Like I’ve been liked to, like I was cheated out of proper service, I feel like I lost money not being able to execute on my trading plan.

In the mean time, my transfer has not progressed. The share sits in Robinhood inaccessible to me. AND all my other shares at Robinhood are also inaccessible.

When I hack it… if I ever try to hack a startup,… I would never be sneaky in this way. I mean, com’on at least leave the customer and investor out of the con!

I hope by the time this blog publishes the company has either changed its ways or else gone out of business. Deploying this kind of unnecessary tactics to retain customers is just bad.

PostScript

Haha, just saw Square unapologetically settle a lawsuite for taking gratuities from caviar customers but not passing the gratuities to the couriers(those who deliver the food) or to the chefs(those who cook them) The company insists that it did not confuse anyone that the gratuity was for Square, the S&P 500 public company and not its small sellers or its individually contributing couriers. (Techcrunch article circa 6/11/2018) There was one time when I made an order, there was a mandatory 18% gratuity. I, as a customer of Square’s customers cursed the courier for demanding uch exorbitant tip… Sorry courior.

This drastically lower my already very low esteem for having provided service to this startup. I’ve increased my trust in Square’s leadership and management in the time that I worked for it. I grew to believe that it wanted o empower the economically isadvebtaged and help he excluded inorities. But it is definitely a company that appreciates hackers and engages in this generalized hacking. Specifically, I am not excluding suggestion that it may take significant advantage of its merchants who are economically disabled or members of an excluded minority. Clearly this law suite is one such case. I feel it has taken advantage of participants of this transaction. What it charged should not be called gratuity.

But, as a shareholder, I applaud, as I did at their bi-monthly townsquare all-hands, the dilligence of their legal team for wordsmithing the legal documents to make this “not Square’s fault”… We gladly applaud it even though it is like applauding a democratic politician’s claim “I did not have sexual relations with that women” (…according to dictionary definition of that word) I guess… I mean, I guess we kind of just accepted that and voted red anyways, right? Yay, let me buy some more when SQ drops at this news.

But seriously though! You all startup hackers should be ashamed of yourselves for taking advantage of those poor drivers! Sheesh!!! I’m going to go out and eat at a restaurant and give the server, to his hands, a proper tip! This is a big karmic dent that needs to be patched with certainty.

Was Axial Japanese as bad as European Nazi’s?

I’ve pondered this question recently. It was largely stirred by a memory of a faint thread in the media or interweb connecting and equating the Japanese massacring Chinese people, such as the Nanjing massacre, to the mass killing of Jews in the Holocaust. The thread compared numbers, and talks of the alliance to rid earth of lesser races.

(Now putting on my duly red blooded American hat) that’s probably Chinese communist propaganda. It’s hard to say this without diminishing the Chinese pain and suffering, but it’s really is possible that Japanese did not try to cleanse Chinese race as the Nazi did Jews. It really is possible that what Japanese did in the Chinese invasion was due to pure greed for land and material and not for some twisted political agenda based on racial cleansing?

I will learn more about the history of my people and the Japanese to understand it better.

It does matter.

The people making the claim that Japanese killing of Chinese is of equal evil to perps of Holocaust obviously has the right to assert that and deserves a proper response–even if they are freaking commies. What the perpetrator thought and believed in does matter as to the degree of the offense.

It has begun

I was involved in a near miss today on I280 northbound traveling from Page Mill rd and exiting on 92. I was driving in the right lane near alpine exit, a large SUV in the second lane from right slowed down to match my speed, and then merged into my lane, slowing down even more. I hit the brake hard and slowed enough for him to veer, very slowly, across my lane into to an exit just a few feat in front of me. The problem was not that this happened. The problem is that the SUV took very measured steps to be in front of me and slow. It was fully in my lane for a good 5 seconds before crossing into an exit lane at the last moment. The problem was not that it looked like he was trying to get me to rear ended him at 65 mph(speed differential was probably 30mph), instead the main problem is that he was accompanied by an identical large SUV that was directly behind me and tailing me at just a few feet.

As luck would have it, I was driving my Nissan Leaf with my daughter. Since I did not have a full charge, I set cruise control at 65 mph, a very slow, but legal, speed for this stretch of I280. My gasoline-free plugin electric car probably weigh only 1/10 of the two SUV that surrounded us. My tiny tires probably has 1/4th the surface area as their combined tire surface. Any kind of collision would not have been pleasant for us by any measure. My slow rate of travel enabled me to slow down and avoid colliding with the erratic driver in front of me. The quick electric motor response probably saved me from being rear ended by the identical SUV behind me.

Fear stirs in me as I think of what happened. Perhaps yet another local organization decided that my blog calling for software engineers union is…antagonizing in some way? It would be pretty convenient if instead I became preoccupied with a severe injury in myself or my daughter…:-(

The QIM: a Measure of Fairness in Servicing

In a previous episode, we discussed the components of QIM, as well as some ways to interpret, measure and perform the decomposition empirically. I suppose it is high time to write out more formally the model.

As common sense dictates, a service of interest, in the context of Fairness analysis, is any system, machine, human or a mixture of both–collectively the Service. There will be an ever present environment, a context, within which the system physically, legally and morally resides–collectively the Situation. The Consumers of this service and a Metric computable from observations. service has a fixed duration of interest to each of its Consumers.

The question QIM tried to answer is: under the Situation, does said Service provide equal service, as measured by Metric, to its Consumers during relevant Durations?

This seems positively silly thing to write about. It is obvious that the local taco shop in my neighborhood does not provide me with the same service as other visitors–those that can order animal guts, brain, ear, tongue, and intestinr stew in Spanish, and enjoy it, where as I, a non-spanish-speaking vegitarian who has never had that type of food cannot order it and choose not to appreciate it at all. The Situation and Metric for this simple neighborhood taco shop seems absurdly difficult to define for it to be a useful exercise. (There are also a homeomorphic Cantonese food, amongst many other foods, for which I cannot say or eat)

As with all human matter, one would expect some compromise when the mind corners itself. For me, this answer is effort. In the course of servicing it’s customers, I would be made happier knowing equal effort was made to service me as was made to service my Spanish speaking neighbors. Why? You ask? Seems like a peculiarly pernicious thing to ask for the pain of your server when the inability to enjoy the food is due entirely to me the Consumer, the service(this, btw is not common, realistically in discrimination situations, the server’s taste is not… so innocent imho), the disadvantaged Consumer seems at fault! To this I must answer, but I paid the same as my neighbors (probably also not true, I had to pay more, but for argument sake, and without any loss, let’s say I paid exactly same price as they) the restaurant should, as a matter of fairness in service, give me the same respect as they give to your other customers. The ingredients and materials you use to prepare the food, the man-hour, the natural gas, the plates they are served on, the dilligence and persistence in the mind and body to produce the final product, these must be no less fore than for my Spanish-speaking neighbors.

In one respect, this approach is useful. Ultimately the effort a Service makes is something under their fuller control than the outcome of the service.
This does not make me happy as I will never enjoy beef tongue, eating it or otherwise, but that is not something the server can affect.

(I’ll surely regret this. I recall a time, maybe a decade ago, on this blog where I said I’ll never floss until the day I cleaned my other end with similarly intrusive externalities… I floss now.)

This does not make me happy as I will never enjoy beef tongue. In a second respect, this demand is made partly out of respect for money. For money to maintain it’s integrity, it’s value, demands must be made at it’s expenditure. For if the money buys me less, then money is worth less to me. Out of my, the Consumer’s, respect for money based market economy Situation, I demand that it buys your expense and exertion. In a different situation, for example as a vegetarian, I may wish to demand that no cow is harmed in making of cow tongue soup, but that is not the more generally applicable economic-driven Situation I am currently addressing.

(And further, often these are stated terms of service using expressions such as “performance in good faith” and “fullfillment by all reasonable/commercial efforts”. Here my stipulation is that the performance will be both in good faith towards accomplishing the service and also not only reasonable, but also equal effort among Customers.)

Therefore, QIM can be applied to this effort based Metric of Service as well.

Trek

Seems like there is a Trek Discovery season II. Wonder what’s left in season I though. So far the folks are still growing into their characters. Captan is stealing the show a bit, honestly, the development of Michael is… rather supernatural. For a human minority, I kind of wanted to see her succeed as a human… but she does I suppose, reintroducing herself to the new security chief…

Picard facepalm to Sarek getting caught preferring Spock over Michael.

Generalization Initialization

I’ve been talking to coworkers about recent batch of papers claiming deep neural networks can or cannot generalize effectively.

I feel I do not have the same respect for this problem as my coworkers. I do not fear it as they do.

Let’s see, how bad could this be?

I suppose an example of this problem is learning to identify a cat. The robot may find out through reinforcement learning that a cat is best identified by scaring it suddenly and hearing surprised meow. So few mute cat exists that accuracy is negligibly decreased by this overfitting. The obvious problem with this is that there are mute cats and Hollywood will make a movie about the one that was used to defeat the AI that overpowered its human creators.

(And the reverse could be true as well, for example toy dogs finding out that scaring children into crying fit is the best way to detect a child from adult)

The intelligent reader will quickly point out that there are plenty of things covered in deepnets-101 that prevents that from happening. (Well maybe not necessarily for reinforcement learning, but straight up deep nets has nice regularizers)

What else could happen? Was there a meme around the internet about the indistinguishability of dogs and fried chicken? The fear is that Cortana would grab the dog and microwave it when you ask it to reheat the leftover from KFC. The generalization in this case is too general—it found anything that could resemble a dog instead of just the dogs. And this was just a meme, not sure if it could withstand serious answers.

More sophisticated problems, often jokingly put on display, are the mistakes that mentally ill people display. Well, mentally ill people and geniuses. The AI could make framing errors: throwing a person into a pool to clean some dirt off of his shoulder. The solution is not within reasonable framing of the problem. But it could be chosen due to the wrong type of generalization.

There is also a problem of leakage. For example, a learning system could overfit training data consisting of FBI profile so much that it is more of a determiner for whether FBI has investigated a person instead of determiner for true crimes. Failure to truly generalize to other populations for whom FBI never collected information is caused by the learning system picking up bias and errors of the whole FBI system consisting of many error-capable humans. The theory, at least for today’s systems, is that it is at least as bad as the human it learns from.

This now indeed seems to be a very interesting problem to consider. But there may not be a one-stop-shop solution to all of AI’s problems. Generalization is probably just one of many things we must solve for in future systems. This is great opportunity for scientific advancement and development of specializations, such as Robopsychology, and psychohistory, and…

But for real.

Mysterious Circumstances

I’ve departed from work under very mysterious and inexplicable circumstances. Just before the Christmas break I was scheduled to work on highly visible project at the company. I wonder if my departure had something to do with my very recent blog post giving president Trump’s engineers as an example of superior group than software engineers? Perhaps someone felt a simile between software engineers to prostitutes was antagonizing to female software engineers? (Technically, that would be due to unconscious bias of the offendee IMHO.)

I don’t blame them. I wouldn’t want to be caught dead employing a unionizer. The company’s hip iconic and meteoric growth persona could be marred beyond recognition by such a scandalous employee! And as shareholder, I probably don’t want a unionizer employer either. It would get in the way of things.

(Don’t mock me a FOB, many many coworkers and leaders mgiht be, but not I. Your coworker and the media of this place would have you believe me that it’s needed but never shouldn’t or needn’t happening here in CA. One helpful friend literally, in these words said:”They’d blacklist you, you know that right? If you complain about unfairness they put you on a list and you will never find a job in the silicon valley again” Although that was a good decade and half ago… andI didn’t believe him then, but now… See here, and here, while they are not exactly anti-union efforts, these tech companies, as decided in out of courts, definitely are messing with selective employment and blacklisting of some candidates for reasons other than their skill fitness! Oh, ahh, my poor SIP, the consequent thought: hey, if they want to do that, Unions would really get in the way of these acts. Unions can detect these things and act to prevent them from happening to the extent they have, and to obtain more just, more remunerative remedies for all affected when it does happen.)

I mean my kids play date canceled by her friends’ parents, my new neighbor starts yelling at me for very very very minor stuff. Five checkout counters at the library, three free, and the people behind me insist on rushing us to use ours. I feel like I’ve been misunderstood by Californian karma for my blog post. I feel so unwelcome here all the sudden.

Honestly though, just because his party aggressively targeted Bay Area industry to tax and pillage, it doesn’t mean we can’t stay within bounds of legality and challenge the administration rationally.

Just because he can be crude, angry, discriminating and unreasoning, it doesn’t mean we have to be. Two wrongs doesn’t make a right. The case against the administration can be made a lot more effectively than they have been made so far.

And it is against California law to unfairly treat employees because of their politics. It is a protected class!

Anyways, all my posts are published long after I write them, so by the time you see this, I may not be here any more. I may be hacked and lose my access to computers all together… I may be in bankruptcy and foreclosures… I might have gone to the west and bought my farm, I applaud you if you hear these primitive mixed metaphors.

I wish the best for everyone still, and seriously hope that we can engage in civilized and legal conversation about politics–even in difficult and poor circumstances.

Finale of Discovery

“…principles are all we have!” declared Michael in the heat of debate with admirer Cromwell about whether or not to blow Qo’nos. What a sharp contrast to “…hope is all we have” exclamation from Star Wars movie.

Yeoh’s costume, btw, reminds me of war lords she’s played in Chinese Wuxia movies. She wears it well.

Sigh, I’d hate to see Philippa summon Michael to see her one last time on her deathbed some years later in the watchers world… like when soong summoned data to bid farewell.

Wow, Ash leaves Michael so that Vog can returns to L’real and take control of Klingon home world… wow what a bitter bitter story for this tortured body and relationships… mind warping to think of the emotions…

And tos theme… I can hardly bear the anticipation…