I am trying to come to grips with the degree of mental retardation that I suffer.

I’ve looked at and pondered the reason for “timed transfer” on Cal-train schedules for the past week as I ride the train to work.

But I just couldn’t figure out why it is arranged the way it is. Below is a picture of the timed transfer between 207 and 211

Basically, it is a slow ride that stops at every station on the 207 to the transfer stations, after which you can transfer to the other train (the 211) and it stops every station from there onward.

So if you had to get from a non-major station in the upper segment, to a non-major station in the lower segment, you have to ride slow all the way.

Would we not accomplish the same with two trains? a slow train that stops every stop, and a faster train that met it half way? that way, if I travel to a minor station in the lower half, I take the faster train to the middle point and then transfer to slower train. If I travel from a minor station in the top half to a major station below, then, I get on the slow train, and meet the fast train half way and ride that to my destination that is a major train in the lower half?

So, to draw it out, make 207′ stop every where, and let’s say it still arrived in the green transfer stations at 6:39. Let’s make the baby bullet 309′ a limited express that stopped at all stations 309 and 211 stops at in the upper half, and arrive at green transfer station at 6:39. In the lower half, 207′ still makes every stop, and 309′ makes all stops that 309 and 207 makes in the lower half.

Then, every one in the minor station above can get to the transfer point, and arrive at at least one of the major stations below. Every one at a major station above, can still get to a minor lower station.

This way, what Cal-train can do is run longer trains (8 instead of 4 or 5) and save some diesel. Surely, two trains 8 sections each uses less diesel than 3 trains 5 each.

I even searched google for the answer, but I just don’t understand it. If we insist on two green transfer stations, it could provide shuttle service…. (since in our scheme we only have one transfer station where as in previous scheme there were two)

In Corporate America, bosses are given special powers so that they can govern their employees. These powers start with the ability to hire and fire employees, as well as to set corporate guidelines for rewarding team and individual performances. In addition, the company has access to personal information about individuals that other individuals do not have access to.

In order to become employed in America, I must give the company my address, my phone number, my Social Security number, my age, my past work experience, what other people thought of me at my last job, etc.

Now, the law specifically prohibits discriminatory usage of this information.

The subject of my blog entry concerns the expanded roll of the company, the executives, and the management. These entities (let’s call them the boss collectively), the bosses has extra abilities over the employees of the company. These abilities include not only the power (not by might, but by right) to control the employees, but also they are given personal information about the employees. The employees cannot take a vacation without justifying it to the Boss. (try it, you will be looked at with different eyes afterwards)

In the internet age and the computer industry, the bosses are especially given the additional power of being able to monitor all electronic communication travelling over the company’s computer equipment. In fact, on the computers, the bosses have, by nature of computer design, administrative superpowers. He can alter bodies of emails or Instant Messages, he can delay the sending and receiving of emails, he can alter your text documents or spreadsheets, and he can change the speed of your computer so that it crashes or pauses at opportune times.

These powers exist. You know it, I know it, and we all know it.

But, what of corporate governance? Has it ever crossed anyone’s mind that this absolute power over employees in the electronic domain is wrong?

And, today, I finally found the problem with the typical reservation: “The company reserves the right to monitor all data travelling over it’s network equipment and to alter these data as it wishes. Bosses have the right to do this because the hardware belongs to the company.”

The problem is that this is not regulated so that the bosses may not comply with discrimination and harassment laws.

For an example, the bosses cannot harass it’s employees by making him “accidentally click” on pornography or any otherwise offensive material just as much as they cannot put up offensive posters that pop up (physically) from the toaster when I go to toast some bread.

Furthermore, the bosses should not be able to impersonate a person’s writing or mistakes. For instance, they cannot alter my emails by introducing typos. The bosses should not be allowed to change the content of my outgoing email in ANY WAY SHAPE OR FORM.

If it sees the need to filter out-going email, then it can do so. And it may choose to inform its employees that this has happened. But the bosses should not be able to change some part of my communication.

Granted, the email box is not a federally installed, government owned piece of equipment. But in general, if I am sending an email, it is generally believed that it is my own person who is thinking and typing the message.

If I make a phone call from work phone, it is believed that I am the one speaking. The bosses cannot introduce sound-bites of swearwords into my phone conversation (and sometimes to the other party only, without me hearing it). Because when I email, and when I call from company, I am not doing so anonymously as “some employee of such and such company”. The person receiving the call and email sees my name, and associates what is said or typed with my  identity. The bosses, by altering my communication violates me, impersonates  me and defames me surreptitiously.

I cannot stress this enough. The bosses cannot make it appear to my friends, co-workers, and my past and future employers that I constantly make typographical errors in every email, should they be allowed to make them hear swear words on the phone when I converse with them, with or without my knowledge. Even if it is on the company’s equipment!!

The reason why the bosses cannot use this power is because they are not able to use it indiscriminatingly. Because the bosses have, and the bosses will continue to let their centuries of hatred based on religion, language, country of origin, sex and hereditary lineage, passed to them from their ancestors run wild in this absolutely unmonitored and unregulated medium for identity and work.

Furthermore, the bosses should not be able to use information gathered from employees’ electronic communication in a discriminatory way. That is to say, it cannot use an Asian person’s email to his wife about vacation plans to make his vacation impossible but schedule it so that white people can go on their vacation. Doing so would be equivalent of establishing an official holiday on all Indian holidays but scheduling mandatory all day long meetings on Chinese holidays.

My blog is but a firefly near a fly trap. It might be hacked… it might be defaced, and it will receive typos…By the time you read this, the sentences may not be sentences any more.

But some where and some time, out there and in the future, on my beloved Internet, I believe that the evil-doers on the grand scale or on the personal level, will be punished, and that truth, justice, and the American way will prevail !!

Okay, this does not need to be said out loud for most people… But somehow, I had to type it out on my blog to understand it…

Well, to start, I should say, the phrase “you’re so superficial” comes to my mind often. By that phrase, I mean it in a most derogatory way. (It’s kind of Chinese I suppose) But sometimes, after some decades of this kind of thinking (and failing), I finally decided to write out what it means to be superficial… When I curse in my head and think of some one as such superficial pieces of shit, what is it exactly that I mean? And am I missing something?

Well, it should be easy. Superficiality often has physical meaning: inside, but near the outside; where inside, outside, and distance are defined. On a person, being superficial often means to make gross and short term observations and to decide or infer based on them. Though, as a description of a person, it is more often used as a judgement of ethics and moral: being superficial is to be superficial in a wrong and immoral way.

I went to wikipedia and tried to read up on morality and ethics and found that if they were formatted into wikibook, the pdf would be some where in the 1.1 kilopages.

Okay, let’s do something simpler instead. Superficiality is a pattern of action and thoughts as defined by superficiality of physical or inferential way. If such a pattern of action and thoughts have negative utility, then these actions are considered Superficial and the actor a Superficial Actor.

As anybody who works in an office knows, other people fart. Often silently but pungently.

A curious question comes to mind. When I smell another person’s relief, what is the right thing for me to do?

My initial reaction, as I did recently, was to walk away swiftly, interrupting my own conversation with the other person, who is possibly the perpetrator.

But this makes me an obvious suspect as the passer of gas. Because, usually, farting precedes or is preceded by or certainly accompanied by the need to have a bowel movement. If I move to leave the room, obvious assumption is that I just farted and am headed away to prevent additional intrusion into others’ noses and to relieve my bowel of it’s contents.

But if I do not leave, then I must suffer the smell until it dicipates naturally, for if I leave anytime before then, or at least, before another person gives up and leaves the room, it would appear that I am in more urgent need of relief than the other person, and therefore the original horn blower.

But if I sit there, silently, disguising my disgust, my obviously artificial facial expression will again make me the trumpeter in my pants.

So, I act naturally. As naturally as one can in such a situation.

If I point out who did it, it would seem that I am attempting to divert attention away from myself. I would only do that if I was actually broadcasting the expiration date.

If someone else point out that I farted, my attempt to deny it would clearly be just another adjustment for inflation.

But if I falsely accept the accusation that I was the one who developed that WMD. Nobody will ever suspect that I was lying!

Sigh, such sad state are my mental affairs.

I never learned database stuff in cs classes… But recently had to use Pig’s Skewed Join and thought it was interestiong:

The skewed join samples the data (by running a full MR through the data) and splits data belonging to the larger keys to several reducers and replicating the smaller table to all those reducers.

My initial thought when exposed to this is that wow, that’s so cool, but so dumb. Why do a full MR to sample the data? Why not utilize hdfs to actually sample records from the input to approximate the split?

Secondly, the optimization to split only the large key into several partitions is unecessary… Consider this instead: Approximately compute the mode hash partition. (i.e. hash key into hash space, compute the hash space that would have received the most keys, possibly exceeding that reducer’s capabilities) And in reality, often this is will not just be identical hash partitions, but identical keys.

Compute conservatively the number of reducers will be needed to compute that hash partition, say this number is k. Then in the mapper stage add to the key a randomly to each record an integer between 0 and k-1. (Add key as in add a separate key, not arithmetic addition). Cross the smaller table with the numbers [0,k-1] and use the integer as the key as well. Join on the reducer side.

I argue that this doesn’t increase, excessively, runtime/resource consumption as compared to the implementation that splits only the largest keys into several reducers. By obviousness. The small keys will be sent to several reducers, but the smaller table will be waiting for it there, so no biggie.

It reduces cost by one MR, because we can sample, or if the data is known, we can just allow the language to specify the number of splits to happen.

r= join x by a, y by b using “skewed 100”;

Finally, to push this to an extreme, Why bother even generating the second key at all? Similar to the Pig’s map-side-join called “replicated”, the replicated join can actually happen on the reducer side. Just send small table to each reducer, and randomly throw records at the reducers (w/o even putting into a bucket), and perform the join there.

The problems only occur when “Small table” is large. In which case, too many different hash keys on one reducer may overwhelm the system. But the proposed system here is no worse than the described system that is implemented righ tnow.

I guess the complication comes in when there are other operations and joins involved. If for instance, an operation on the join reduces the size of the data significantly (i.e. filter) then putting it on the mapper side is worth the while. Because, obviously.

But if that is not the case it seems always more worth while to replicate to reducer than to mapper. Because there is a chance that the small table won’t have to be replicated as much (big hash partitions will only receive one row in the small table). AND, the cost of communicating that extra bit of random integer is almost surely smaller than copying smaller table to mappers, joining, and then taking big table row + small table row and sending to reducer. Right?? Send smaller table once to reducer and be done with it. Send a small integer along with big row, but that’s probly compressed away because it’ll be the same integer in most rows. (Recall that data is skewed, so most data is in the “mode” key and that key all goto a few reducers, so “mode” key and it’s random number all get compressed to nil)

So, I guess the solution is not to only to allow us to specify the skewness but also to specify whether

Using “replicated”

is map side or reducer side.

Using “replicated to mapper”

vs

Using “replicated to reducer”

I want to explore the exact situations under which the above proposed features are superior. As well I’d like to extend the analysis to multi-table situations.

wlog:
Join A by key, B by key, C by key…

with sizes A>B>C….

what kind of key distribution will…..

I wish I had spent more time in biz school learning the trade. When you have a b2b company (My former employer Bittorrent, for instance) how do you perform market analysis? Who’s the target market?

Well, it seems that for a b2b company, the target market is some times better defined by the target market of the target market.

Using Bittorrent again for example. The target market are numerous. (The product DNA, Distributed Network Accelerator)

The target market is anybody who wants to transfer the same bits over the network.

But this is made easier if we analyze the target market of the target market.

Who needs to transfer same large files to many people? (a priori, not streaming as is the case with youtube, google earth, … hmm actually, maybe that’s another target market)

The users who can withstand some latency:

• Movie watchers downloading HD/blueray movies.
• Game players downloading game graphics
• Sysops downloading linux DVD images
• Scientific data sharing
• Intra-company transfer of data file.

From there, the target market suddenly becomes clear

• Media companies: youtube, hulu, porno sites, netflix, if p2p streaming is supported then all the better
• Game designers or Game distribution channels
• Linux distro companies or organizations (redhat, etc., etc.)
• Scientific data sharing is usually between university labs (so market to the universities directly and get a share of the NSF/NIH funding)
• Intra-company is harder, there are of course products in this area from companies such as Microsoft.

So because we begun to investigate the target market of our target market, the target market analysis suddenly became easy. And the sales prospects just became really really obvious.

Of course, most successful marketers will have done this subconsciously and arrived at the finally analysis directly as if by pure genius. For me, I’ll just have to square that target market analysis process.

For quite some time now, I’ve been considering getting myself cosmetic surgery. Today, while at work, some thing subtle finally pushed me over the edge and I spent a good lunch hour googling and day dreaming about what to get myself.

I made a list of big stars that I might want to look like. The girls at the office thought I was just being cute and making conversation, but I hadn’t the heart to tell them that I really feel that I need it.

So here’s the list… We started with

George Clooney,
then

Recently the governator announced that he will sign AB 183 into law. I mentioned to my co-worker that this will create a micro-bubble and raise the price of house by $10k for the duration of the tax credit (may1-dec31 2010)

Today, he comments to me that actually the bubble might be more than $10k higher price. Let’s make the simplifying assumption that instead of a tax credit spread through 3 years, the tax credit is actually a cash payment that the government gives you at the point of closing. And also, let us assume, without serious loss of generality, that the home buying reward is given to all home buyers instead of new home buyers.

Then, he’s argument is, if you put down the $10k that government gives you into the downpayment, you actually save more thank $10k. Because you won’t have to pay the interest of those $10k. So the total  savings will be

10000*(1.05)^30 = 43129.42

(assuming 5% mortgage interest), so a dumb person may be persuaded to buy a house as high as $43129.42 above it’s actual market price if there wasn’t an incentive.

Now, let’s take this a step further. Say the interest rate is zero, and the lowest amount of down payment a person can pay in order to buy a house is 10%, then this $10k incentive allows him to buy a house that is $100k more expensive than the market price without incentive. So gives us a bubble of $100k above normal.

The psychology of this is well known. A person would probably be satisfied with keeping $5k of the incentive and allow seller to take the other $5k, but he would probably not buy the house if he was only able to keep $1k and the buyer takes $9k.

Anyways, the true size of the bubble will be known soon enough…