Signal and Noise

Shortly after midnight on September 26, 1983, Lieutenant Colonel Stanislav Petrov stood between the analytical pronouncements of Cosmos 1382 in the Oko system and global nuclear war that might certainly annihilate civilization. Paranoid Soviet leaders had a system built to alert to a preemptive attack. It used infrared sensors to detect the hot gas plume of an incoming missile. When it was fooled by the setting sun into predicting one and then five incoming missiles, it was a computer programmer who recognized it for the mistake it was. For this we can all be thankful.

Algorithms often process clean input data well enough to get a useful answer with minimal or no rework. It’s when we follow our data blindly that we get into trouble. This is the very real danger of an algorithm-driven world. That shouldn’t completely dissuade us from using algorithms to help us make decisions. But given the potential for poor input data (either through inconsistent measurement or incomplete knowledge of input state assumptions), it’s best to keep a human in the loop in critical systems.

Recently, I was investigating death rates by state for different types of cancer. Two tables in Cancer Facts and Figures 2015 had just the right data to put together – new cases (p5) and deaths (p6). All I needed to do was cut and paste the data into Excel and make a new sheet to divide deaths by new cases. How hard could it be?

Not so fast. Cut and paste from the PDF didn’t work. So, I printed a page of the original document, scanned it and saved the resulting image as a PNG file. Abbyy FineReader Sprint 8.0 converted this image into a nearly perfect Excel spreadsheet with only a couple of cells to fix. That result was good enough, but there was still the nagging issue of the printing and scanning. Why waste the paper and ink when you could just read it from the page electronically?

With a Mac, I used Grab to capture a TIFF image of the page. The attempt to convert this image with Abbyy created a nonsense spreadsheet. I saved the TIFF image in the PNG format and converted it. Now the rows and columns were captured properly, but 10% of the 561 cells (51 rows and 11 columns) had errors. This kicked off an analytical odyssey typical of those pursued by scientific investigators working in a new area with bad data. You know it’s going to end badly but hope drives you forward.

The scanned version previously recognized had been captured with a text method that sharpened and added contrast. I replicated this with the color adjustment tool in Preview (again this is all using bundled Mac tools). Increasing contrast to 75%, only 40 errors were seen. At contrast of 100%, errors were further reduced to 30. Sharpen at 100% was roughly equivalent. With a combination of contrast at 25% and sharpen at 75%, I was down to 11 errors.

Still though, this was disappointing compared to the nearly perfect work made of the printed version that had been scanned. Meanwhile, Abbyy was calling attention to this with every attempt to read the file. It persistently displayed a warning, “The image resolution is too small, please re-scan with the resolution greater than 150 dpi.” Now the time to listen to the warning was at hand. The scanned image had been captured at 150 dpi. The TIFF generated by Grab was 72 dpi. The answer was clear.

A quick scan of the Internet suggested that best practice for scanning resolution was closer to 300 dpi (Abbyy, Fujitsu, Univ of Illinois). Could I get 300 dpi image from the PDF file? It turns out that the Save As function in Preview can save a PNG formatted image of a selected page in a PDF file. An input box lets you enter any numeric value for the resolution without argument. Nice, but no matter what you enter the image is saved at 150 dpi.

Fortunately, that was good enough. I had a worksheet with death rates by state for selected cancers, and the exercise was sufficient as a lesson for this article. But that brings us back to the threat of nuclear annihilation instigated by a closed loop monitoring system. When one builds a mission critical system, can you trust it enough to run without human intervention? What if there was a threshold that was good enough, and we were 99.999% sure that it told the truth as we knew it?

Let’s see… Maybe I want to scan tables from multiple PDF files. Could I use my method unattended? Look, there is package called Xpdf with an extractor called pdftoppm that scans at 300 dpi. It runs from the command line. I could raise some venture money for a big data / analytics business, scour the Internet for PDF files and start extracting. What could possibly go wrong?

| Comments closed

Displaying Census Data

This post should give you an idea of how census data can be displayed geographically using choropleth (an area that marks property on a map) outlines to communicate insight derived from analytical investigation to a broad audience.

Every ten years, the US government makes a thorough counting of its citizens. With an estimated population of over 310 million people, the only questions posed to everyone are those necessary to identify the population uniquely.

But businesses and policy makers need more data, on more subjects, updated more frequently. As a result, the U.S. Census Bureau also rigorously interviews approximately 250,000 people each month. The answers are then aggregated and reported as the American Community Survey (ACS), in 1-year, 3-year and 5-year collections.

Depending on the questions posed, one might integrate the available private data with state, county, city, zip code or census tract detail to get answers. Presenting such results geographically is often the best way to make a point.

TIGER/Line Shapefiles from the Census Bureau describe these detail areas. They are designed for use in geographic information systems, and provided in the shapefile (.shp) file format. We prefer to present visualizations in web-standard browsers using the D3 (Data-Driven Documents) library, which requires conversion to geojson and then topojson formats.

This D.C. Choropleth example shows 2012 District of Columbia census data displayed by census tract relative to other tracts. Home owner occupancy rate presents well, as it is relevant to everyone in the census tract and normally distributed. An overlay shows points of interest in and near the different tracts.

Hopefully, you take away from this example the basic use of the choropleth in the presentation of information geographically. It’s a work horse for stories that rely on demographic and/or economic data.

| Comments closed

D3 Visualization

The d3.unconf (“a one-day gathering of data visualization practitioners”) is coming Mar 29, 2014. I can’t wait. When it comes to exciting big data visualizations, so many are being built on the D3 (Data Driven Documents) library. There is great beauty in the investigation of data. D3 lets the data scientist share that beauty with others.

This should be a cool event because it asks participants to prove their level of commitment by posting an example when they register. This is a nice bit of genius that will surely raise the quality of the event. Those of us who attend will already have shared something that we learned, proving that we are already in the spirit.

My example came from a seed planted by Ian Johnson. I wanted to share a visualization at the Jan. 28 meeting of the Bay Area d3 User Group in San Francisco. So, what would be of interest to the group? Ian said something to the effect, “Tell us something about ourselves. There’s a list of almost 2000 visualizations. What’s in the data?”

The result was D3 Authors by Domain. I used the d3.pack layout function to organize nearly 1500 identifiably authored visualizations by the web domain where each had been presented online. The result was beautiful, almost fractal. The link shared above points to a block that presents the image along with the behind-the-scenes notes of how it came to be. This five minute video captures the joy of its discovery, as I present it to they d3.bayarea() audience.

Please look for me at the upcoming d3.unconf if you plan to attend.

| Comments closed

Mission Needs A Goal

Your mission is what you do every day. Where the vision statement dares to dream, the mission statement is direct and tactical. Unfortunately, so many organizations make a hash of the two. They waste days and months arguing over replaceable weasel words. Avoid this. Combine the two into a single Big Hairy Audacious Goal statement, and give your team a vivid and exciting dream to chase.

The mission statement tells employee and customer alike, what value will we deliver every day through our products and services. It answers the question, how will we achieve our vision? Let’s look at what can come out of a rigorous group effort to define one.

Guided by relentless focus on our five imperatives, we will constantly strive to implement the critical initiatives required to achieve our vision. In doing this, we will deliver operational excellence in every corner of the Company and meet or exceed our commitments to the many constituencies we serve. All of our long-term strategies and short-term actions will be molded by a set of core values that are shared by each and every associate.

Yikes. An epically bad statement such as this one takes an enormous number of staff hours to complete. Only a very large Fortune 500 company can afford such waste to produce so little value. Startups and fast growing companies don’t have time for this. Even if tempted to use a free mission statement generator (Laughing Buddha, Corporate B.S Generator, or MSG, bureaucratese has no value in any business.

Better to write one Big Hairy Audacious Goal statement that combines a vision of the future with the reality of your product. You want something like this powerful, unique statement from Twitter,

give everyone the power to create and share ideas and information instantly without barriers

or this one from Google,

Organize the world’s information and make it universally accessible and useful.

How can you get the benefits of writing your own statement without messing it up? Here are five simple steps.

  1. Identify your best writer, and give that person the job of mission journalist.
  2. Have the writer interview employees and customers, and listen carefully to the answers. This person will pose who, what, why questions to the people who build and use your products. Sources like The One Page Business Plan, Co.Design, Fast Company, and Entrepreneur) offer particular queries that can uncover everything you need to know about what makes you great.
  3. Have the writer craft a finished statement in the form of a Big Hairy Audacious Goal. Ask them to keep it short and use powerful, vibrant words. Each element of the mission must count.
  4. Share the statement with the team. Let it resonate in their minds. Capture their response through both survey, and unsolicited comments.
  5. Acknowledge and respect the team’s feedback with a deliberate, careful revision, and move forward.

In summary, let your staff and customers tell you what makes your company special. Find the common thread and turn it into a Big Hairy Audacious Goal. Amplify it with high quality, exuberant writing. Then post it where everyone can see.

Stewart Noyce writes and presents on innovation strategy, a passion he has fed over a multi-decade career. He continues to accept intense, hands-on product management consulting assignments on new product teams. Visit TruNorth Consulting for more details.
| Comments closed

Vision Invents the Future

If you want to predict the future, invent it.  It’s not enough to have a new idea, the successful innovator must envision the future.  Imagine how the world will be changed, for the better.  Write it down.  Then engage others and share your core values with them.  Even if your vision sounds wildly ambitious at first, it will become more real every time you talk about it.  Think of it as your Mt. Everest.  It might take you many years to climb to the top, but nothing is impossible.

When herding cats, it’s best to have a bowl of milk.  Literally, that can mean milk and cookies.  Back in the day, Sun Microsystems had beer bashes on Friday afternoon.  One would stand around the keg with Scott McNealy.  Apple had a live band, a wine and cheese spread, and bowls of Oreo cookies with milk to dip them in.  You might imagine which ‘bash’ had more appeal. After an obligatory plastic cupful someone would say… “Sorry Scott, I have to go now.  It’s been nice and all, but I have some more coding to do tonight.”

A quarter century later, the food perks trend has reached New York startups.  Though it’s not really what attracts great talent to your company.  There are two more important things.  First, they want to know that their work will have lasting and significant value.  They want and expect to change the world, which makes them very interested in their employer’s core ideology and values.  Second, they know they have a short shelf life.  Anyone in their twenties should work at 5 companies over 10 years.  Unless a company they join quickly rocks the market there is no reason to stick around and vest worthless shares.

Satisfying both needs, the company worth serving has an audacious unifying vision.  This vision expresses the values of the company that each person hired will share.  This vision describes broadly how the company will change its market (and very possibly the world).  This vision creates a measuring stick for a set of goals that are otherwise impossible to quantify five years in advance.  Like a nice big bowl of milk, the vision focuses everyone involved on a common goal (customers included).

Think of great vision as a landmark used for navigation.  The landmark is your goal, a destination.  Set your sights on it.  Remember to look up at it periodically.  As the winds and currents throw you off course, reset and adjust to get back on track.  A great vision can be attained over many years, but not without constant strategy iteration.

So watch where you’re going.  Look ahead.  If you want to predict the future, create it.  Do remember though.  If it’s not audacious, no one will care.  Go big or go home.

Stewart Noyce writes and presents on innovation strategy, a passion he has fed over a multi-decade career.  He continues to accept intense, hands-on product management consulting assignments on new product teams.  Visit TruNorth Consulting for more details.
| Comments closed

Inciteful Headline, Insightful Copy

When you want to get noticed, you whisper.  There was a time this was true.  When E.F. Hutton talked, people listened.  But they had an attentive audience.  If you are not already massively popular, this strategy just isn’t going to smash through the mindshare clutter of the Internet.  What can you do?

You can try click bait.  Write inciteful social media headlines so enticing and alluring that no one can resist the urge to click through.  Catch the attention of the surly, sleep-deprived and impatient.  Lead them to the rich trove of content they desire. Just follow this super easy, simple 20 step process for writing effective headlines (with pictures).

Well, it’s not that easy.  Professional or amateur, marketers must engage busy people inured to spam and weasel worded pitches.  Readers seek knowledge, and new information that satisfies their curiousity.  They use every spare second of each day to catch the stream of content that flows through the world wide web.  Finding truth in this stream can become an obsession.

The seekers want insightful copy.  They seek novelty and idiosyncrasy.  Starved for quality, they demand individual voices speaking with clarity and authority.  Instead, much of what they read is written by bots, or people writing like them.  Feeding from this trough of gibberish shoehorned into popular, proven memes is like binging on potato chips.  The reader can’t stop because they haven’t been satisfied.  Is it any wonder that people walk into meetings and family dinners where the humans are face down in their phones.

As with addictive games and mainstream magazines, an occasional stroke of genius flashes across the wire.  This content is devoured like jumbo shrimp wrapped with thin slices of kobe beef.  The author and source are now worth following, or bookmarking, or adding to a reader list.  How was this amazing piece discovered?

David Ogilvy said, “five times as many people read the headlines as read the body copy.”  That was when print advertising ruled the earth.  The headline brought people in to read the long copy that closed the deal.  In our Twitter world, the tweet may not replace the headline but it does draw one to the page that holds it.  The headline still matters, but now we can attract more potential readers through serendipitous interactions.

With chance meetings in mind, double down on the provocative tweets.  Flog the idea from every angle.  Great incitement salves the pain of the downtrodden, and enrages the trolls.  It tweaks hidden desires, the ones normally kept tightly locked up.  It attracts attention, and keeps it by piquing curiousity.  People know their desires.  They know much less the strategy to achieve it.  Convince them that only one click separates them from the one extraordinary insight that will change their life forever.

But don’t waste their time.  The consumer seeks the truth, and has scarce time for the quest.  If your brand disappoints enough times there will be others to replace you.  if the content is inadequate or irrelevant it is better to say nothing than to say anything at all.

| Comments closed

Love Them All

At the family dinner, a middle-school boy admits to his first crush. His father says, ask her to the dance immediately. The sooner you tell her your feelings, the sooner you will know if she loves you back. Love them all, and find the one that loves you back.

The same principle applies to ideas. The moment something enters your mind, write it down. Test it later. Even if it is less than brilliant and sent to the scrap heap immediately, at least you didn’t obsess on it for a week or more.

Confirmation comes from Yves Behar in San Francisco magazine, April 2013 (cover photo). He wants his team to sketch out their ideas as soon as they have them. It’s the quickest way to weed out the bad ones.

“If you never draw a bad idea, it will remain in your head. The best way to purge it is to draw it”

| Comments closed

Advice for Small Business

Bob Nelson seeks to train small businesses around the world, bringing livelihood to the poor. His request, what are the top 10 pieces of advice you would give to a small business just getting started?

I offer this list. Appropriate to groups of any size in any market, it calls attention to the universal issues of competitiveness and value that must be present for success. There are no surprises here. However, you must be on top of every item in this list to have a chance.

  • Growth Market – Participate in a very large emerging market that is growing quickly.
  • Market Blocker – Solve the nasty problem that prevents this market from growing as fast as it can and should.
  • Compelling Value – Offer a unique value proposition so complete and compelling that customers demand immediate delivery of your product / solution.
  • Many Customers – Target segments where you can start with at least 10 customers and quickly grow to hundreds.
  • Core Competence – Do well what you do well, particularly what you do better than anyone else.
  • Sustainable Advantage – Follow a strategy that puts you in a stronger position relative to competitors with each passing year.
  • Accelerated Transactions - Make it easy for your customers to find and pay for your product / service.
  • Gross Profits – Make money on every transaction, and quickly cover your fixed expenses.
  • Sufficient Equity – Accumulate adequate equity capital to launch the business and fuel its growth.
  • Great People – Surround yourself with high quality people who complement your skills, and hold onto them.
| Comments closed


Just as you can’t make a patient healthy by bleeding them to death, one can’t make a business healthy by simply laying off employees. Recently, HP announced that it would reduce staff by 29,000 (CNN) with the goal to simplify business practices and accelerate innovation (BGR).

Massive layoffs are a sign of bad management (Forbes), the same kind of quackery that bloodletting is to medicine. The best profit making and revenue creating ideas are likely to come from those people who work directly in the trenches. But HP isn’t picking its most innovative thinkers to stay, it’s selecting by age through early retirement. Given an even distribution of talent across all age cohorts, as HP eliminates older and more expensive workers it will be cutting the very people it needs.

HP is now a dead man walking.  At Sun Microsystems in 1990, I heard Bill Joy say, “No matter who you are, most of the smartest people work for someone else.”   Increasing the number of smart people that will work for someone else is not The HP Way.  After years of growth through acquisition, HP leadership must no longer believe that employee brainpower is the company’s most valuable resource.

| Comments closed

Obama or Romney?

Neither Barack Obama nor Mitt Romney impress me as candidates for President of the United States.  In Presidential Debate 2012, I said that I was looking for evidence that either had a summary plan which included a long-term vision, a day-to-day mission and concrete objectives.  From the transcripts, word counts for the Oct 3 and Oct 16 debates show their primary interest to be jobs (104 matches) and plans (99 matches).  Vision, mission, goals, objectives and strategy barely receive a mention.

Here’s the summary:

Candidate President Barack Obama Governor Mitt Romney
Vision Uses the word future 18 times, and tells Jeremy it is bright. Says Romney has a vision that government creates jobs, which Romney denies. Comments once on the President’s vision, using the opportunity to dismiss Obama’s failed policies. Uses the word future 8 times, and wants to make it ‘bright and prosperous.’
  • Keep the American people safe
  • Create ladders of opportunity
  • Protect life and liberty through military force
  • Maintain our commitment to religious tolerance and freedom
Objective Mentions jobs 45 times in two debates. “what I want to do is build on the 5 million jobs that we’ve created over the last 30 months in the private sector alone” Mentions jobs 57 times in two debates, “I was in Dayton, Ohio, and a woman grabbed my arm and she said, I’ve been out of work since May, can you help me?.”
  • Invest in education and training
  • Develop new sources of energy in America
  • Change the tax code to help small businesses
  • Reduce the deficit in a balanced way
  • Make North America energy independent
  • Open up trade with Latin America, and crack down on China
  • Make sure our people have the skills to succeed, and the best schools
  • Get us to a balanced budget
  • Champion small business

So, I’m disappointed. Neither Obama nor Romney used the debate to paint a clear picture of the future they wish to mold. They now it’s coming. They are sure it is bright. But what is it exactly? Well the candidates either don’t know or don’t want to say. That makes me nervous.

Despite their shared objective to add jobs, and similar strategies, they have competing plans that ensure partisan gridlock in 2013. You can tell they don’t plan to work together by their unwillingness to quantify the number of jobs they expect to create. The root of the country’s current malaise is big business and big government arguing over how to reallocate the income of US citizens.  They’re rearranging deck chairs on the Titanic.

| Comments closed