Scalability Problem of the Day: Neural Networks

Deep Learning is the next frontier in computer science. After some initial breakthroughs, scientists and engineers are running into a major scalability problem: increasing the number of neurons doesn't improve a neural network's performance.

(MTR, 5/21/14) We found that if you put a lot of GPUs [specialized graphics processors] together we could make a much bigger neural network—10 billion nodes, with 16 machines instead of 1,000.

We used that same benchmark [images from YouTube videos] that the Google team did. But even though we could train a much larger neural net, we didn’t necessarily get a better cat detector. Right now we can run neural networks that are larger than we know what to do with.

This is a typical situation in Silicon Valley (I described in an earlier post). We are at a point where Machine 1 (exponential growth in computing power) is ahead of Machine 2 (Applications). Most likely, the next S-curve, i.e. a new growth cycle, will begin within the next 5-7 years.

tags: problem, scalability, constraint, machine1, machine2, silicon valley,

Trade-off of the Day: Revenue vs User Experience

Yahoo reported 2012 Q4 results that show a company stuck in a 15-year old trade-off between selling more annoying banner ads and improving user experience:

Display ad revenue excluding commissions to other websites in Q4 fell 5%, even though the amount advertisers were willing to pay for ads rose 7%, the company said in an earnings statement.

...the fall in Q4 was also at least partly an effect of Mayer's mission to improve the user experience on some Yahoo properties, including its home page, wrote Cowen and Co. analyst John Blackledge, who rates Yahoo as neutral. Fewer ad impressions in Q4 were "partially a result of an attempt to rein in site clutter and improve user experience," he said.

Yahoo's business model is bound by the Revenue vs Quality trade-off. In contrast, Google provides relevant text ads that improve user experience AND generate more revenue. The difference in business models creates a huge performance gap.

 

tags: tradeoff, business, internet, model, constraint, 

Trade-off of the day: Liquidity vs Growth

In "Corporate Finance" textbook I found this fundamental financial trade-off,

The more liquid a firm’s assets, the less likely the firm is to experience problems meeting short-term obligations. Thus, the probability that a firm will avoid financial distress can be linked to the firm’s liquidity. Unfortunately, liquid assets frequently have lower rates of return than fixed assets; for example, cash generates no investment income. To the extent a firm invests in liquid assets, it sacrifices an opportunity to invest in more profitable investment vehicles (p. 32. McGraw Hill, 2002).

The textbook treats the trade-off as a given, but from business practice we know that successful investors, such as Warren Buffet, break the trade-off by using insurance "float" and/or borrowing against fixed assets. Access to low- or negative cost credit seems to be the most creative way to deal with the problem.

tags: trade-off, constraint, finance, business, problem, solution

Quote of the Day: Investing in Globalization.

There’s no way to invest in a world where globalization fails.

The question then becomes what are the best investments that are geared towards good globalization. Facebook is perhaps the purest expression of that I can think of.”

- Peter Thiel.  (quoted from The Facebook Effect, by David Kirkpatrick.)

tags: investment, model, business, trend, constraint, startup

Enabling constraint: single width axle (China 3rd cent BCE).

Standardization of transportation was one of the long-lasting legacies of the Qin dynasty in ancient China. Ideology-wise, standardization had roots in legalism, a doctrine of universal application of clear laws; the doctrine developed in the 4th century BCE by Shang Yang, the prime minister of Qin. After Qin defeated the majority of other Chinese kingdoms and established a unified empire, they undertook enormous efforts to transform the state, including building advanced road infrastructure.

A 4,700 mile network of roads was developed to ease travel in the empire and to the frontiers. Regular staging posts allowed horses to be changed frequently and provided places to sleep at night. The single width of cart axles encouraged trade as there was no delay in moving goods through the provinces. All carts would travel in the same wheel ruts and there would be no need to change to a cart with a different axle width in different provinces.

Standard roads and single width axles were enabling constraints because the helped develop commerce. On the other hand, standardization of thought imposed by Qin turned ugly. They ordered and enforced destruction of all books that were in conflict with their ideology. Moreover, because at the time thought was largely transferred through oral teachings, Qin buried alive hundreds of Confucian scholars, who disagreed with Legalism.


Contrast this with Facebook, which established a standard way of communicating between people (social utility as Zuckeberg calls it) but allowed for free expression on top of the utility.

tags: constraint, payload, distribution, facebook, social, network, infrastructure

Intel's new Atom: a shakeup in the mobile market?

Intel wants a piece of this action:

Note how processor design that was key to Intel's dominance in the PC and server space turned out to be a major long-term disadvantage in the mobile segment.

Dec 21, 2011. MTR - Previous Atom designs spread the work of a processor across two or three chips, a relatively power-intensive scheme that originated many years ago in Intel's PC chips. But now Intel has finally combined the core functions of its processor designs into one chunk of silicon.

tags: innovation, constraint, mobile

Enabling vs Disabling constraints.

An excellent cartoon showing the difference between enabling and disabling constraints.

 

(c) The Economist

Often, constraints are perceived as limitations, i.e. "strings attached." But most of the time, constraints work as infrastructure elements supporting a myriad of routine processes of everyday life. For example, a traffic light limits our freedom to drive through a crowded intersection, but at the same time it enables us to move safely in traffic.

tags: constraint, example, course

A strategic false belief in trade-offs

In his seminal paper "What is Stragegy?", cited 4367 times on Google Scholar and included into every business strategy textbook, Michael E.Porter writes:

But a strategic position is not sustainable unless there are trade-offs with other positions. Trade-offs occur when activities are incompatible. Simply put, a trade-off means that more of one thing necessitates less of another. An airline can choose to serve meals - adding cost and slowing turnaround time at the gate-or it can choose not to, but it cannot do both without bearing major inefficiencies.

Anyone who flew on a commercial flight recently can attest that the latter part of statement is false (italic, bold - ES). Airlines do serve meals AND they make money on this service by charging for the food. They also charge for luggage and other conveniences, more than compensating themselves for the inefficiencies.

U.S. airlines collected $3.4 billion for checked luggage last year, according to a government report issued Monday. That’s up 24 percent from 2009 and a big reason the industry made money again after three years of losses. In 2010, the major airlines made a combined $2.6 billion in profits, less than they collected in bag fees.

The trade-off once considered fundamental by the top business strategist turned out to be a false belief into what can or cannot be done - a simple lack of imagination. People believe in trade-offs because they are taught to believe in them. Once somebody figures out a way to break the trade-off - puff! - the whole business strategy based on it falls apart.


tags: trade-off, strategy, business, barrier, constraint, soft, trend, innovation, breakthrough, model, quote

The oldest (and the dimmest) lightbulb in the world

Recently, the city of Livermore celebrated the 110th year of continuing service of a ligh bulb:

For more than a century, the 4-watt "Centennial Light" -- believed to be the world's longest-burning bulb -- has hung from the modest rafters of Fire Station No. 6 in Livermore.

4 watts! By design, this old bulb produces at least 10 times less light than an average modern one. The key reason for that is the limitation imposed by the low-current DC electricity distribution system invented by Edison. His main business constraint was the cost of copper wires: the higher the current, the thicker and more expensive the wires would have to be to feed the bulbs. But expensive wires would not allow Edison's system to cover a large city area, making it a poor competitor for gas lighting installations popular at the time.

Therefore, Edison settled on a low-current system, which required high-resistance light bulb filaments. Unfortunately, a carbon-based filament found by Edison, the famous bamboo one prominently shown in books and movies, would heat up during operation and burn out fairly quickly. Furthermore, the carbon filament would eventually blacken the inside surface of the bulb, turning it even dimmer. To make his system more reliable, Edison had to limit the brightness of bulbs used, the bulbs bright enough to beat competition from candles, kerosene and gas lamps. Only with the invention of high-resistance tungsten filaments by Hungarian engineers in the beginning of the 20th century and, later, the process of filling the bulb with an inert gas, light bulbs began approaching levels of brightness comparable to today's standards.

Eventually, the world created highly resistant bright light bulbs that consumed more and more electricity, the electricity coming over long transmission lines from distant power plants burning fossil fuels, losing lots of power on the way and in the bulb itself because most of the power went into heat. Paradoxically, with low-power LEDs and local solar panels producing low-current DC we don't need the heat and the transmission lines, but we still lack efficient power storage systems. When these are deployed at scale, we'll go back to Edison's good old idea of a 4-watt bulb, which, implemented with new technologies, will shine brighter and greener(!) than ever before.

tags: system, evolution, power, distribution, tool, constraint, trade-off, payload, course

Invention of the day: electric car starter.

There's a class of inventions that improve original breakthrough technologies to such a degree that the breakthroughs reach the mass market. In the beginning of the 21st century, invention of blogging democratized web publishing by helping people who didn't want to learn HTML create their own web pages. Even now, most blogging services, including blogspot and LiveJournal I use regularly, offer two editing modes, HTML and direct. With the latter, you don't need to know any HTML tags to orchestrate your text so that the browser understands how do display the content when it pulled from the server to your PC or mobile. The removal of the user skill constraint enables explosive growth.

 A hundred years ago the same thing happened to the automobile when Vincent Bendix invented an electric starter for gasoline engines. The starter is a miniature electric motor coupled to the main gas engine to crank up the engine from a dead stop. Today, you only need to turn on the ignition key or press a button to open a flow of electric current from the battery in your car to the starter. The starter turns and, being linked mechanically to the rest of the system, turns the crankshaft of the engine.
 
Before the invention, the driver had to do it manually. That is, using a hand crank, the driver had to crank up the engine - a feat that not only required significant physical abilities, but also the willingness to get down and dirty with mechanical tools on the road. (Just like a web designer would have to learn HTML codes, scripts, frames, etc. to put together a passable web page.) People who eschewed the mechanics of automobiles had to either hire a chauffeur or stay away from the automobile market altogether.

Introduction of the starter eliminated the problem, especially for middle class men and women. The horseless carriage, which most people at the time perceived as an expensive toy for men, became a mass market product that drove American innovations for more than 50 years. Consumer loans, the highway system, the oil and gas industry, tire manufacturing, suburban homes and other features of the modern life we love and loath are direct consequences of the automobile revolution. I can't say that Vincent Bendix of Memphis, Tennessee started it one hundred years ago, but he definitely contributed to the process.


tags: invention, problem, technology, improvement, industry, infrastructure, innovation, system,  constraint

Time to resurrect privacy? At least for children.

Last spring, by the end of the Principles of Invention (BUS 74) class I taught at Stanford CSP, a group of students identified an online privacy and security as one of the more important problems to address in the near future. A recent Zogby poll sponsored by Common Sense Media confirmed the students' assessment:

...three out of four parents say that social networks aren’t doing a good job of protecting kids’ online privacy. The poll finds that 92 percent of parents are concerned that kids share too much information online, and 85 percent of parents say they’re more concerned about online privacy than they were five years ago. The Zogby International poll also finds that 91 percent of parents think that search engines and social networking sites should not be able to share kids’ physical location with other companies until parents give authorization.

A large part of the problem is that people have very little knowledge and control over how the information gathered by social networks is being used and where it ends up eventually. Somehow, we've created an environment where others know a lot more about you than yourself. But, unlike a relationship with the doctor, teacher or lawyer, this personal information asymmetry is governed neither by trust or law. It is as if every time you talk to a friend somebody is eavesdropping on your conversation. Phone companies are prohibited from doing this, but social networks are not.

tags: control, information, social, network, detection, constraint, niche construction

Another argument for early education

Puberty worsens children's learning abilities. Unfortunately, our education system was created over a hundred years ago, when kids, on average, reached puberty later in their lives, and very few of them could afford high school. As a result of changes in human biology and culture, we've got schools that teach languages and science (which, arguably, is just another language) at the worst possible time: during last years of junior high and first years of high school.

When children hit puberty, their ability to learn a second language drops, they find it harder to learn their way around a new location and they are worse at detecting errors in cognitive tests.

Why is this? Sheryl Smith and her colleagues at the State University of New York now reckon that all of these behavioural changes could be due to a temporary increase in a chemical receptor that inhibits brain activity in an area responsible for learning.

On the other hand, teenage boys learn very quickly how to play complex computer games. It's quite possible that the academic learning model doesn't fit their natural learning abilities.

tags: biology, education, information, system, constraint, trend, biology, information

Remarkable how the US still pays millions of dollars for a military technology that can be defeated with a $500 laptop and $25.95 worth of software.

WSJ. WASHINGTON -- Militants in Iraq have used $26 off-the-shelf software to intercept live video feeds from U.S. Predator drones, potentially providing them with information they need to evade or monitor U.S. military operations.

The Air Force has staked its future on unmanned aerial vehicles. Drones account for 36% of the planes in the service's proposed 2010 budget.

Today, the Air Force is buying hundreds of Reaper drones, a newer model, whose video feeds could be intercepted in much the same way as with the Predators, according to people familiar with the matter. A Reaper costs between $10 million and $12 million each and is faster and better armed than the Predator.

I would think that in the nearest future video feeds from police drones will be easily intercepted by hightech criminals.

From a 5-element analysis perspective, this is a very good illustration of how critical Payload packaging is for the overall system integrity and performance. The ability to handle the format of the video feed in question is deeply embedded into all relevant subsystems. Changing the format would require a technology overhaul that would cost the military billions of dollars.

tags: five element analysis, payload, control point, control, information, drones, transportation, 10X, constraint

Dec 17, 2009. Bloomberg reports on projected shortage of doctors in the United States:

Last year, there were 16,721 fewer primary-care doctors than needed in inner city and rural areas, according to the U.S. Health and Human Services Department.

The doctor crunch is a result of an aging population and a rising demand for specialists, according to the federal health department. By 2025, the nation as a whole will confront a shortfall of as many as 159,300 doctors of all varieties, said Ed Salsberg, director of the Center for Workforce Studies at the Washington-based medical college association.

A graph from a 1954 article by Ernest Jawetz in Annual Review of Medicine looks remarkably similar to the Gartner Hype Cycle "discovered" in 1995. Both graphs have an enthusiasm peak, a disappointment pit, and a productivity plateau (see below).

Gartner Hype Cycle (courtesy wikipedia.org)

With Twitter we are probably still in the early stages of the cycle (see Google Timeline snapshot):

tags: innovation, cycle, diffusion, pattern, theory, book, infrastructure, niche construction, constraint

Time to imagine a world where battery power for portable devices is no longer a problem:

Scientists at the University of Missouri are developing a small nuclear battery that they say can hold a million times more charge than standard batteries.

The radioisotope battery, being developed by Jae Kwon of the Department of Electrical and Computer Engineering and other researchers, is the size and thickness of a penny.

The battery is designed to drive micro/nanoelectromechanical systems (M/NEMS). Such devices include labs on a chip, and biological and chemical sensors.

You can't recharge a nuclear battery, though. Only replace.

I continue listening to Yale' netcasts on Entrepreneurship. Today, my "guest speaker" is Jordon Goldberg(J.G), CEO of Stickk.com , a web service that enables people to make commitments to a variety of causes, the major one being weight loss.

Key learnings:
1. A clear distinction between invention and innovation.
The talk is almost entirely about the business of innovation, i.e. making a startup work, product development practices, marketing, networking, and etc. The original invention by two Yale professors serves as a background and media attention grabber. The public is fascinated with the idea, which is a good indicator that the problem that stickk.com addresses is very important for the society.

2. The difference between short- and long-term planning, which is measured in constraints and trade-offs, rather than time.
J.G. is acutely aware of his personal and corporate constraints, such as money, communication distances ( world is not flat!), staff interactions, and etc. For an early stage innovation company the time is almost always "now", therefore it is very dependent on earlier strategic decisions.

3. He mentions the impending move into the corporate services market, which would be a good topic for a 10X diagram exercise. The exercise would explore ideas related to the increasing scale of each target market (person, family, community, company, town, a major corporation, government, global company, etc. )

4. The VCs' aversion to encumberences.
VCs don't want to have any, even potential, committments to research, non-profit, and other goals that do not relate directly to money-making. This is a good indicator that they anticipate management (control) problems, which may arise out of non-functional resource expenditures.

On a more abstract system level, problems might stem from future resource commitments, i.e. "bad constraints". On the other hand, forming money-making opportunities, e.g. via networking, is a "good constraint".