I’ve been thinking about autoactivations recently. This is one of those great innovations that stood up to the test of time, it still works after a lot of debugging and exposure to new data and models.

I find that I have been referring to autoactivations as pre-activation because they occur to deep neural nets before the parameters are actually mixed with input data (or previous layers activations) but if you look at the two expressions:

1. To pre-activate a parameter means to apply nonlinearity before it is used. e.g. preheating the oven, the suffix is a verb and happens before something else.
2. But a pre-activation is actually an adjective meaning before any activations. e.g. pre-trial motions. It’s suffix is a noun and becomes the subject to be preceded.

And actually similar problem applies to ante- prefix. So, to avoid confusion, we should probably refer to autoactivations as foreactivations and to foreactivate the layer. This prefix also means before and it works both for nouns: foresight, foreknowledge, forethought, forerunner, foreword, foreman, and also works for verbs: forecast, foreshadow, foredone, foreshorten, forewarn, forestall, foredoom. In each case the suffix is always the prior thing before but never preceded by another.

So, let us all try out foreactivations and related approaches. The speed up in training will surely be a good thing for humanity, at least, we won’t be consuming as much energy training models without foreactivations.

😀

I just read a rather alarming article about Asian Men harassing Asian women for dating and marrying white and other non-minority races. Admittedly this is alarming because I may have expressed jealousy towards some white kids having more tofu from Chinese girls than I. And also, some of them seem to know more about Chinese culture than I. I think casting aspersion on the motives of Asian women becoming embroiled with non-minority races has certainly crossed my mind a million times if not materialized into blog entry. (It will before this entery publishes)

Personally, I don’t feel it unethical or illogical to have these thoughts. I’m not terribly ashamed of them either. But by the same token, a lot of the envy come from seeing some very very very successful friends and acquaintances make Asian-women-non-asian-men relationship work. There are really nice best-of-both-worlds marriages starting with TWO ceremonies(parties)… Two homes on different continents to take new family members to… Two successful cultures, religions, languages, And those beautifully exotic hybridized kids… Love them to pieces! So much so that the feeling really hurts some times.

But that is not to say harassment is right. Asian women probably suffered more than non-Asian women throughout history. But today they deserve the respect of a fellow human being. We, Asian men should never harass Asian women for their independently made decisions!! And incidents of harassment should of course be treated with urgency.

Asian men do have a hard time in America, as the female author of the harassment piece notes. Although it sounds a little bit belittling, these weaklings who cannot compete reproductively resorts to emailed harassment of lost mates. I mean, god, as an Asian men, reading this piece makes me hate Asian mankind even more than I already do, having all my inferiorities and inferiority complexities. It really doesn’t make me respect Asian women more when she publicly writes these belittling pieces about a group of people whom she chose not to associate with.

Yes, just like women have to fart and poop on rare occasions, I believe non-minority women are capable of racist behavior too. (Not speaking from personal experience, of course, every women I know are not racists, but in the large there must be enough racism to make the pairing challenging) Therefore the celebration of those few successful Asian-men-white-women relationships should not be disparaged. The lack of celebration for Asian-women-non-asian-men relationships is absolutely no reason to diminish the success of those who ventured and succeeded.

But I am a bigger person.

I think a rational concern, if I were elder to an Asian girl, would be the balance of power. Yes yes yes, there’s love, but a little balance never hurts. Perhaps one thing that Asians don’t understand about each other is what we think of each other’s power in a relationship. Who decides where to live? Who decides what to eat? Who decides what to drive? Who decides when to have kids? Who decides how to spend money, Who decides …

Asian men may imagine that in mixed relationship, the girl lacks power, that she give up rights and freedoms, those she would enjoy in a racially equal relationship, for … whatever it is that she does it for with the non-minority, maybe better sex? Asian women, probably think they have all the power in the relationship. I mean if she doesn’t feel safe, she probably wouldn’t enter into that relationship (there, I am a bigger person, I respect rationality of girls when it comes to love)

The disparate perspectives probably causes all the angst of the article. So, IMMHO, can Asian girls prove they have access to their rights and privileges? Can the girls show us, and the world that they are in an equal partnership? And Asian men who can’t get Asian women, can you try white girls, I’m guessing they can be quite nice to you if you find your way around surrounding racism and into her heart. My guess is that women of all races are feeling and thinking rational beings.

Ugh, random rambling, hope this note will still make sense when it sees the light of day