Random Thoughts on Randomness

Here's a random thought on randomness...

In a typical state lottery, like California's Powerball, a player chooses five or six combinations of numbers between 1 and 59.

So, how likely is a lottery's winning set of numbers to be 1, 2, 3, 4, 5 or 1, 2, 3, 4, 5, 6?

Surprisingly, it's no more or less likely than California's most recent Power Ball winning numbers: 3, 13, 17, 42, 52, 24. Random numbers are random numbers. While 1, 2, 3, etc doesn't seem random, it's no different than any other combination with non-patterns. Don't forget, since we're dealing with pure numbers there's solid mathematics behind it.

Pulitzer Prize for Coding and Blogging?

I've debated whether coding is art.

Writing prose has a lot of similarities to writing code. Both activities require a lot of time spent inside a text editor. The key difference is the final product. When writing prose, the audience sees the final written letters. When writing code, the audience sees what the software does, not what it is in its raw form.

Coding seems more like a craft than an art when you consider that it's one key part of software engineering. This difference is even more pronounced when considering the Pulitzer Prizes.

The Pulitzer Prize board usually awards a prize in each category to a single person. Yet, there were a lot of people on the team who contribute to the winning book, news report, or editorial cartoon. Compare that to making movies or software which require large teams. Software released today is not written from scratch, like a book or poem. This is obvious when you consider the OS and code library dependencies.

Blogging, on the other hand...

The Question is Begged

Why is there no Pulitzer Prize category for blogging?

I wholeheartedly believe there should be a Pulitzer Prize category for individual blogging. After all, Pulitzer awards their prizes to individuals. Some of their prizes are for journalism and some are for art. Are the Pulitzer's about the content or the medium? Meta-blogs have won Pulitzer Prizes, such as the Huffington Post. But I would no longer consider Huff Post a blog, like, say, TechCrunch. Rather, HuffPo is an online journalism news source. There's a distinct difference.

Bloggers are doing important work. The Pulitzer Prizes should formally recognize this with its own category. When it is, I shall nominate the Scripting News blog. Not just for being around for 20 years, next month, but rather for defining what the true essence of blogging is.

If you agree, then please let the board of the Pulitzer Prizes know:
pulitzer@pulitzer.org

Why is it Called Coding?

This piece is a response to this morning's Facebook and scripting.com post, What "coder" means and why it's bad by Dave Winer. I originally wrote this as a Facebook comment and then posted it here on Mea Vita: Carpe Diem.

Dave, I read your Facebook post and scripting.com piece, What ‘coder’ means and why it's bad. I think you and I are seeing eye-to-eye.

Some might say we’re splitting hairs, but this distinction is important since good definitions make for clear ideas.

The terms coder is too generic to describe what we do. The problem is similar to a painter. The guy who paints my living room is a painter; and so is Frida Kahlo and Pablo Picasso. The former has to paint inside the box, the latter outside. While we also describe the latter as artists, that term also has problems since it's too broad.

What's the Difference?

Coding is what we do when we write code and it's what a writer does when s/he writes prose. But we are not coding when we design a database, develop an API, or architect a server farm. These tasks are more than coding, they’re engineering since we’re synergistically engineering software. The sum is greater than its parts.

I think all software engineers are coders, but all codes are not software engineers. Perhaps programmer and coder are closer synonyms. The problem with the term programmer is it's not specific enough. Programmers can also be DJs who strategize radio station formats. In an archaic sense, programmers used to also be people who set up manual programs to sell large amounts of stock (this is different than computer guided program trading).

Some companies put programmers in restrictive boxes by only allowing them to code to spec. It’s an assembly line that lacks innovation and squelches initiative. It's much like the painter I hire to paint my house who has no say in the color or pattern.

Coding just happens to be the most visible part of software engineering. It’s dynamic, mostly linear, and more tangible than an API or database schema, so that’s how the layperson describes it. Plus, it's a single word. It's much catchier to say, "I'm coding," vice "I'm software engineering."

PS – I noticed that I have a coding category on my blog, but I don't have a software engineering category. That's changing with this piece.

Monitoring How Things Fail

As our world becomes more complex, things fail in ways we never expected. In the military, we trained for different scenarios so we had documented responses. I see much less of this scenario based training in high tech simply because it's too complex to cover critical, unimagined, failures. Grace Hopper captured this sentiment best when she said, "Life was simple before World War II. After that, we had systems."

This week I started reading a book I heard about on NPR, The Checklist Manifesto. The author, Atul Gawande, is an accomplished surgeon. He noticed that seemingly small, yet critical, bits of information were overlooked in the operating room. Borrowing from the experiences of airplane pilots, Dr. Gawande began using checklists before operating on patients resulting in fewer mistakes in the OR.

When I originally launched Adjix I encountered different ways that servers could fail. A few incidents stick out in my mind.

Don't Backup Onto Your Only Backup

Always have a working backup. This seems obvious. I noticed one of my Adjix servers had slow disk I/O – in other words it seemed that the hard drive was failing so I backed it up onto a backup drive. Unfortunately, the backup never completed. I was left with a failed server and an unstable, corrupted, backup copy. The important lesson I learned here was to rotate backups. Nowadays, this should be less of a problem with services like AWS.

A couple years later I saw a similar issue when I was consulting to a startup that maintained two database servers in a master/slave cluster. The hard drive on the master server was full causing their website to go down. Their lead developer logged into the master server and started freeing up space by deleting files and folders. In his haste, he deleted the live database. When he logged into the slave database he discovered that his delete command had replicated which deleted his slave database. Their last offline backup was a week old. He was fired as the rest of the team took spreadsheets from the operations and sales departments and did their best to rebuild the live database.

How Do You Define Failure?

Apple uses WebObjects which is the web app server that has powered the iTunes store and the online store since they were created. WebObjects included one of my favorite RDMS, OpenBase. The beauty of OpenBase was that it could handle up to 100 servers in a cluster and there was no concept of a master/slave. Any SQL written to one server would be replicated to the others within five seconds. This is very handy for load balancing.

OpenBase's process for clustering was elegantly simple. Each database server in the cluster was numbered, 1, 2, 3, etc. Each database would generate its own primary key, 1, 2, 3, etc. These two numbers were combined so that the number of the server was the least significant digit. For example, database server #8 would generate primary keys like 18, 28, 38, 48, etc. This ensured that each database server's primary keys were unique. The SQL was then shared with all the other databases in the cluster.

Here's where something looked better on paper than in the real world. If one of the servers failed, it would be removed from the cluster. The problem was, how do you define failure?

If one of the database servers was completely offline then that was clearly a failure. But, what if the hard drive was beginning to fail – to the point that a read or write operation might take 20 or 30 seconds to successfully complete? Technically, it hasn't failed, but the user experience on the web site would be horrible. One solution would be to set a timeout for the longest you'd expect an operation to take, say five seconds, and then alert a system admin when your timeout is exceeded.

Who Watches Who?

When I launched Epics3, I had to monitor an e-mail account for photo attachments. I used a Java library that implemented IMAP IDLE which is basically an e-mail push notification standard. Perhaps there was a limitation in the Java library I was using, but IDLE simply wasn't reliable in production. It would hang and my code had no way to detect the problem. My solution was to simply check the mail server for new e-mail every ten seconds. This was a luxury I had since my bandwidth wasn't metered and Gmail didn't mind my code frequently checking for new e-mail.

Like Adjix, Epics3 was a WebObjects Java app. WebObjects uses a daemon, wotaskd, that checks for lifebeats from my app. If the app stops responding, wotaskd kills it and restarts it. The problem I had is that my Java thread would sometimes hang when checking for new e-mail. The app was alive and well, but the e-mail check thread was hanging. The solution to this problem was to have the e-mail check thread update a timestamp in the application each time it checked for new e-mail. A separate thread would then check the timestamp in the app every few minutes. If it found that the timestamp was more than a few minutes old, the app would simply kill itself and wotaskd automatically restarted the app. This process worked perfectly, which was a relief.

Things don't always fail as we imagined so it's important to avoid a failure of imagination.

Unable to Decode Playground Data

I've been coding in Xcode's Playground using Swift for the past two months. It seemed that my code was touching on an Xcode edge case causing it to stop evaluating with an error message: "Error running playground. Unable to decode playground data." I can tell where the problem is since the playground sidebar stops displaying output at the line of code that's choking. But I can't tell what the problem is.

The line of code having the problem is a function call at the end of a do..while loop. I initially thought my string manipulation was causing the issue since Swift strings are a little different than the Java NSString that I'm used to in WebObjects.

Narrating One's Work

I figured it might help if I wrote about my issue. Perhaps someone else is having the same problem. A quick Google search shows that a few people are encountering the same issue. But too few are having this problem to find a definitive solution other than chalking it up to an ongoing Xcode bug.

Almost There

I initially thought I had discovered the cause, earlier today, when I changed the half-open range operator to a closed range operator (i.e. I changed ...  to ..<). Once I made that change my playground compiled all the way to the end. But this was a short-lived victory when I restarted Xcode and the playground error returned. Toggling between the half-open and closed range operators at least gets my code to compile and run in the playground. So, perhaps I'm getting closer.

Swift First Impressions

Hiking the Pacific.

Earlier this month, Apple announced a new programming language, called Swift. It's designed to be fast, safe, modern, and interactive.

Swift is big news since it was unexpected. I haven't been this excited to learn a new programming language since Sun released Java in the mid-1990s.

Java and Swift both took about four years to develop. But, Swift is already more mature than Java 1.2 was two years after its initial release. Sun used to have an office down the block from the Apple Campus at Mariani One. When I started working at Apple, in 1998, we joked that we could hear the Java API's deprecating across the street in the Sun building. I do not imagine many Swift APIs deprecating anytime soon since they're based on, and bridged to, Cocoa's Objective-C APIs. (Coincidently, Java 1.2's codename was Playground which is the same name of Swift's interactive coding environment.)

Getting My Feet Wet

Last week I created a couple simple Cocoa Swift apps for both OS X and iPhone. It was a piece of cake. To this day, I still love that I can drag between my code and my UI in Interface Builder to link up ivars (outlets) and functions (actions).

I spent most of this past Saturday getting up to speed on Swift. I read the docs and watched a few tutorials including the Introduction to Swift, Intermediate Swift, and Swift Playgrounds videos from WWDC 2014. So, after hiking half a dozen miles along the Pacific, yesterday morning, I decided it was time to dig deep into Swift.

What to Code?

I wanted to write an algorithm requiring a fair amount of trial and error without much mental heavy lifting. For me, the answer was string parsing. I once spent a long time coding, testing, and debugging Java (Eclipse with the WOLips plugin for WebObjects) to parse SMS text messages for newspaper classified ads:

My Tweet Storm Swift Playground

Sell  (Item name)  ($Price)  (ZIP code)  (Item details)

For my first real Swift task I reverse engineered Dave Winer's Little Pork Chop algorithm which lets you send out a tweet storm. Basically, Little Pork Chop breaks up blocks of text longer than 140 characters into tweet size chunks.

Swift and Powerful

In a nutshell, Swift's Playground is worth its weight in gold. The Playground's interactivity is powerful. Typically, I write a few lines of code, compile it, and then run it. I would never recompile an app after writing every single new line of code. But that, effectively, is what Playground does for me. It kept me focused and my code clean. I caught my syntax and logic bugs in real time as it displayed my variables and loop counts. Even better was that my infinite loops were immediately visible in the sidebar. I can see myself writing all my algorithms in Swift's Playground before copying them to my projects.

Swift Playground Gotchas

On the flip side, it doesn't seem possible to step through code, so loops execute until they terminate. Once a loop completed I could see what happened by examining the value history or using Quick Look.

Quick Look into an array and then into each element.

Another wrinkle I encountered was trying to get the character at a string index. The best solution I came up with was fairly ugly:

var currentChar = String(Array(subTweet)[indexOfCurrentChar])

This might be due to the fact that an index into a string array element, which is usually equal to the number of bytes, doesn't hold true if you're using UTF encoding where each character may require multiple bytes. If that's the case, then I need to write my own String indexer. Please let me know if I'm off base.

Free Code

It took me about two and a half hours to code and debug my tweetstorm algorithm in the Swift Playground. Am I getting Swift? Yes, I most definitely am, but I'm probably still using old coding patterns while I learn the slick new modern Swift syntax.

One final piece of Swift beauty is that you can download my Tweetstorm Swift Playground and run it for yourself in Xcode6-Beta.

To Offshore or Not to Offshore at Apple?

When I worked at the Apple Online Store we were organized in teams of six.

My team consisted of four software engineers, one project manager (Scrum Master), and one QA engineer. The QA engineer was a permanent part of our team. He was a white-box tester. We wrote our own unit tests and demonstrated scalability with our component tests while our QA engineer verified our logic. He'd look for obvious issues like uncaught null pointer exceptions while digging deeper in search of ambiguous cases like poor security implementations.

New Code Here

Any issues he found in code we were working on in the current sprint were fixed. The rest of the bugs were entered into Apple's bug tracking system (Radar). Once a week we'd meet to prioritize bug fixes. We off-shored the bug fixes to India since it wasn't sexy work. Once it was fixed, we reviewed the code and verified it before integrating it into the main branch.

Bug Fix There

Offshoring bug fixes worked beautifully. Each bug was clearly documented: what really happened vs. what was suppose to happen. I had no idea who was on the other end fixing our bugs but I realized they were intelligent and hard working. However, I could tell they weren't experienced with our technology (WebObjects) or conventions. I've seen the offshore team hard code SQL queries directly into Java. Other times, I've seen objects instantiated simply to access static Java methods.

The beauty of offshoring a bug fix is we wouldn't have to revisit it when it was corrected. Contrast that with coding new features which is under continuous development.

New Code There

Since offshoring bug fixes worked so well we decided to give them a shot at new development. We quickly discovered that was a mistake. The offshore team didn't have enough context to write good code. Their implementations were too brittle.

This problem frequently happens in any coding organization that's offshoring a new development. Without a product roadmap, the offshore team simply writes code to do exactly what you asked for; and no more.

I've never seen specs sent to an offshore team to refactor code. That would be too nebulous of a task. By the time I'd document all the ins and outs of a spec I could have written the code myself. The real problem is that we don't know how our code will behave until we run it.

And that was the crux of the problem with the code written by the offshore teams I've dealt with. They could only do exactly what you asked for, now, without knowing what was coming.

Software engineering isn't an event – it's a process. It's a process of continue improvement and refinement. It's iterative.

Author: Joe Moreno

$5,000 Security Breach, Part 2

Every so often I write a blog post that immediately receives many thousands of views. Part 1 of this story fell into that category.

Where I last left off, on Thursday, I was in the shower when I had an epiphany. I had figured out how my Amazon Web Services credentials were compromised. At least I suspected, but I was running late, after my call with Amazon, as I got ready for the Spring Fling tech event. I didn't have time to comb through my public repository account so I deleted my entire GitHub account. I had only used it once, years ago, when I checked in an open source WebObjects project I had developed.

Jodi Mardesich interviewed me for the details and gave my story a great write up at ReadWrite.

Coda update: Amazon has confirmed that they'll grant me a one time exception for my faux pas.

Author: Joe Moreno

Lazy Programming

There are two types of lazy programming, good and bad.

Good Lazy

Lazily instantiating and populating data structures is a perfect example of a good design pattern. This technique is also how CDNs populate their edge servers. Don't create or store anything until the last possible moment.

When implementing this technique, I use accessors that have the same name as my instance variables (ivars). Below, my _employees ivar is set to null when a class is instantiated and it's not populated until the first time it's touched (accessed). This is the beauty of key-value coding accessor methods.

private NSMutableArray _employees = null;

public NSMutableArray employees()
{
    if (_employees == null)
    {
        this.setEmployees(new NSMutableArray());
    }
    return _employees;
}

public void setEmployees(NSMutableArray newEmployees)
{
    _employees = newEmployees;
}

Depending on my performance requirements, this design pattern would work if I needed to save memory. However, if memory isn't an issue, but, rather speed, this might not be an ideal solution since each time the employees() method is called there's a an O(1) test performed to see if the private ivar is null. In cases where speed needs to be optimized then it's best to pre-populate the data structures (caches) before the web app begins accepting requests. At the Apple Online Store, we pre-populated only when necessary. In every case, though, the key is to avoid premature optimization.

Bad Lazy

The goal of a software engineer is to provide the best possible user experience (BPUX).

As a programmer, I'm not shooting for perfection but I know when something can be done better. (If I went to sleep last night then I had time.)

If I have to code something that's singular or plural I'll go out of my way so it doesn't read:
You have 1 item(s) in your cart.

It's not very hard to code:
You have 0 items in your cart.

You have 1 item in your cart.

You have 2 items in your cart.

There is no shortage of websites where I've entered my phone number (760.444.4721) or credit card number (4111-1111-1111-1111) only to hit enter and been told I made a mistake and my digits need to be reentered with only numeric characters.

Some programmer had to go out of their way to search the string I entered, confirm there was a non-numeric character, and then return an error message to me. This is my big pet peeve – it's too in-your-face. I entered all the information the programmer needed and they could have parsed out the digits. When I'm coding, I simply write a cover method to return only the numeric digits.

Software engineers aren't sales people, so they don't live the ABCs.

Writing Words, Writing Code, Hemingway Style

One thing that motivates me to write is reading great writing. Whether I'm writing words or writing code, the ability to capture an idea and write it in an impactful way is powerful.

Hemingway – much like Apple – knew how to pare away the cruft to get to the heart of the experience.

When I first began writing fiction I read Hemingway's short stories for inspiration. The first one I read was The Snows of Kilimanjaro where he vividly described a scene without explaining the details.

...he and the British observer had run too until his lungs ached and his mouth was full of the taste of pennies...

This description hit me like a ton of bricks. In this single sentence I understood Hemingway's writing style. When someone's shooting at you the adrenaline deep in your throat tastes exactly like copper pennies. Hemingway had seen combat – he knew what adrenaline tasted like – so there's no need for him to explain it.

The West Wing: "You tasted something bitter in your mouth.
It was the adrenaline. The bitter taste was the adrenaline."

A reference to the bitter taste of adrenaline is also seen in an episode of The West Wing. Josh Lyman is in denial about his PTSD after being shot. A Yo Yo Ma performance triggers a PTSD episode and a psychologist jump starts Josh's counseling session by telling him about the bitter taste.

Hemingway left out details which pulls in the reader rather than shuts them out. That's hard to do. And Hemingway knew exactly what he was doing which he described in his essay, The Art of the Short Story:

A few things I have found to be true. If you leave out important things or events that you know about, the story is strengthened. If you leave or skip something because you do not know it, the story will be worthless. The test of any story is how very good the stuff that you, not your editors, omit.

Writing workshop

It's pure chance I came across Joyce Maynard's writing workshop, last spring, that lead me to her home in Mill Valley to work on my writing. There's nothing better than being taught by a woman who's earned her living as a writer her entire adult life. I'm writing this piece, today, because, yesterday, she pointed out that even the best writers have to handle rejection. And it's through Joyce I feel a connection to Hemingway since she lived with J.D. Salinger and Salinger met with Hemingway during WW II.

Oversimplifying Simplicity

The Way to Eden.

I'm reading Ken Segall's thoughts and experiences while working with Steve Jobs. He's had so much interaction with Steve while at Apple and NeXT that he's a cornucopia of best design and marketing practices.

Segall talks about how "one" is the simplest of concepts. It's an intriguing philosophy – there was even an entire episode of Star Trek dedicate to this concept and its followers.

This belief in "one" is why Apple's mice, track pads, iPhones, etc., from the beginning, have only one button. One is where it all begins.

What's the simplest numeral system? It's certainly not base 10 (decimal) since you have to memorize 10 different digits. Is it base 2 (binary)? After all, computers and human DNA work in binary (ones and zeros or A-T and C-G combinations) for storing information. Certainly binary is the simplest? Au contraire; how many people can convert 1010 from binary to base 10? Not simple... not simple at all.

The simplest numeral system.

It turns out that unary is the simplest numeral system for representing natural numbers – in other words, unary uses just ones. Before there was written or spoken language this is how a cave man would keep track of "How many?" things he owed. Take a pile of rocks and for each one of something, you move a rock to another pile. Do the reverse when taking inventory.

This is how a bouncer counts people at the door or how the simplest of card counters tries to beat the house at blackjack. We've all used unary to keep track of things when we tally items with four slashes and then a diagonal.

Something Simpler?
Where I disagree with Segall's thinking is when he points out "zero is the only number that's simpler than one." Ironically, this not the case as I learned from my assembly language professor, Mr. Lee. If you think back to when we learned Roman numerals in grade school (I, II, III, IV...) you'll quickly realize that there was no numeral for zero. This is also true in other ancient civilizations' numeral systems such as Chinese and Arabic. As simple as zero seems, it's a fairly complex concept to have nothing of something – just try to ask any handheld calculator to divide by zero and you'll see that it does not compute.

Trying to be simpler than the simplest makes things more complex.

Technical Interviews: The Missing Piece

Typical Interview Question: Write a Java method to reverse a string.

TechCrunch had a recent piece about the demise of the technical interview. In software engineering, the technical interview involves writing code. Companies like Amazon and Google have a reputation for asking brain-teasers such as, "How many gas stations are in the US?" or "How many Ping-Pong balls can you stuff into a Boeing 747?" The idea is to see the job candidate's thought process. While these questions are mentally challenging, it's probably not the best indication of how good the candidate is at programming.

I've been through a number of software engineering job interviews where I've been asked to write code and discuss fundamental computer science questions. Writing code is an important part of hiring software engineers and it definitely has its place in the job interview process. And, it's perfectly okay for the candidate to make typos or have syntax errors when writing computer code on a white board. The idea is to see if the candidate understands the fundamentals of computer science such as Big O notation when it comes to a binary tree [O(log n)] or hash table [O(1)] or the basics of recursion and language syntax.

The Missing Piece
One thing I've noticed missing from all my job interviews over the past 15 years is that no one has ever asked me to show code that I've already written, refactored, and trusted for many years. The beauty about reusing code that either I or someone else has previously written is that code you don't have to write is code that you don't have to debug.

Software engineers who live for and love writing computer code have many side projects. You'd be hard pressed to find a good software engineer who doesn't have something currently deployed whether it's a web application or smart phone app. Just like an artist has to paint, or a poet has to write – regardless if they're paid or not – a coder has to code.

The Alternative
The current software engineering interview at a decent tech company involves a series of 45 – 50 minute long interviews where a pair of employees ask the job candidate questions. This process can last four to six hours and the key part that's missing, today, is where the job candidate gets to show off what they've previously written and released. This is especially important for a 40+ year old job candidate who should have a massive bag of tricks since they've probably been coding, on a daily basis, for more than a quarter of a century.

Instead of multiple 45 minute interviews with two employees and a job candidate, it would be much more effective to have a couple 90 minute interviews with four employees where the candidate can show how they architected, coded, and deployed a website or smart phone app. Ideally, the candidate could ssh into their live servers to show the details, challenges, and architecture of how a web app works while showing off the code that he/she has written to accomplish it. Writing code on a white board is very academic; seeing code that a candidate has deployed and maintained over several years is about as real as it gets.

No company would hire a graphic designer without seeing the job candidate's portfolio so why don't tech companies demand the same thing from software engineers?

The Art of Coding at Any Age

May 26, 2013 update: Dave responded to this post with a podcast.

I've written many lines of code sitting here, where I wrote this blog post.

Yesterday, Dave Winer shared a New York Times interview with Billy Joel where the Piano Man said, "I thought there was a mandatory retirement age at 40, but then the Stones broke that barrier."

Dave was born the same year as Steve Jobs, Bill Gates, and Yo-Yo Ma. That means Dave is well past 40 years old which was thought to be the age when computer programmers (coders) were brought out back and shot. For most programmers, a successful career in Corporate America means up or out. You start off as a software engineer, then you become a tech lead followed by forgoing coding to manage direct reports.

People like Dave and I, who enjoy the trenches of coding past 40, are not the norm. A 30 year old software engineer looks upon a fellow programmer, 20 years senior, as out of touch. (Except, in rare circumstances, when they're looked upon as one of the greats who works close to the kernel meaning that they really know what they're doing.)

While Dave has successfully created and sold businesses, his first love is programing. A couple days ago Dave made a point that if he were a visual artist or musician then no one would bat an eye at the fact that he's closing in on 60 while still coding every day. No one asks, "Why is Yo-Yo Ma still playing the cello? Why hasn't he moved on to conducting?"

Elegant Code
Dave also asks, "Why can't people see that this [coding] is an art?" That's a very good question and I love Hugh MacLeod's comment, "Art’s purpose is to express consciousness." While software usually serves a process purpose, we still write it much like we'd write most anything else in a text editor such as a novel. But is it fair to consider it a form of art?

What is the purpose of art? To create. To inspire. To express one’s self. To make one aware of one's surroundings. To make life better, etc. There's no simple definition. To me, holding most any modern Apple product feels like holding art, albeit a highly functional piece of commercial art.

The problem with code, as a form of art, is that people don't see what was created, rather they only see what it does – its function. Code has function. Art has design. Code is a means to an end application. Fine art focuses more on aesthetics than utility. I cannot think of a form of art where the work created is not what's seen which is different from a computer program that gets translated into machine code. Consumers of code only see the interface, not the implementation. Again, this seems very different than, say, a movie since the general public critiques and studies the film, not how it was made.

Art is displayed or performed. Where would people observe code? In its raw form or its final application form? One could argue that an Apple Store is a museum for displaying products of art like a gallery, but the challenge with code as an art is it doesn't exist without a medium. While that's technically true of, say, a poem, I can still hold O Captain! My Captain in my hands in its final form.

The creative process of programming is definitely more art than skill, much like writing a story, suitable for someone to do at any age. Perhaps the ageism in high tech is due to the ever changing technology as more senior programmers stick with older, more comfortable, systems? But, in software engineering, like rock and roll, perceptions will change regarding age.

Chances are, though, if you're still coding on a daily basis, into your 50s, using current technologies, then you're undoubtedly very, very good. To Dave, I ask, "Who else, past 50, codes like you?" Maybe it's time for a Museum of Computer Programming with Ada Lovelace and Grace Hopper inducted into the hall of fame.

The Future of Coding

You don't have to write computer code for very long before realizing that you're going to spend a lot of time inside a text editor. And, a good chunk of that time is going to be spent debugging.

In many disciplines, there are only a few fundamentals. In boxing, there are only four basic punches: jab, cross, hook, and upper cut. In computer execution, like DNA, there are only two states (zero or one; or A-T or C-G, respectively). It's amazing what can be created with just two possible states mixed up in endless combinations.

In computer science, most programs continuously do three basic things:
1. Declare (foo = bar or NSTimestamp now = new NSTimestamp());
2. Test (if now > later then a=b or if snow > 6 then school = "closed")
3. Jump (GOTO 500 or JMP SIGNON)

Which begs a question that I've been pondering. Centuries from now, will computer programming still involve editing text? After all, programming is just a static form of communications between man and machine (pardon the sexism, I'm going for alliteration).

For many millennia, humans have primarily communicated statically, through writing. There are obviously other forms of communications, such as speaking and non-verbal communications (just stare at a stranger for a long time to communicate "I am creepy.") But other than writing (to include images), it's not really fixed (without the aid of technology, e.g. video).

So, just like building a house has always involved assembling small things like bricks and boards, will coding still involve text editing in the 23th Century?

The Single Most Important Step to Securing a Web Site

I've worked as an employee and consultant, in a technical capacity, for tiny companies through multi-billion dollar corporations. With the exception of the companies where I've been a founder, I've noticed that the technical teams at these companies would have a hard time immediately detecting an attack or abuse unless it caused obvious damage to the functioning of their website.

For small to medium size websites (< 5-10 requests/second/web server) the single most important defense against attacks is to continuously monitor logs. The simplest way to do this is to "tail" logs for all instances of the web server, app server, and security logs (ssh, DB, etc) 24/7 on either a system admin's or developer's computer. It would be great if this could be automated, but, many times, it's going to take a human to notice something abnormal.

Most abuse/attacks on a website begin with probing the website itself (port 80/443) or via SSH (port 22). Once abuse or an attack is noticed from a centralized source, the first step should be to block that IP address and then contact the ISP of the attacker/abuser ideally with a phone call to their network operations center or via their abuse@example.com e-mail address which is usually listed when conducting a whois lookup. In either case, the offending ISP will ask for a copy of your server logs which should be cleansed of any third party data. In other words, only send the offending ISP a copy of the logs that specifically pertain to the attacker/abuser.

Typical web server logs should contain at least the following info:
Requester's Host Name/IP: ec2-75-101-189-255.compute-1.amazonaws.com
Timestamp with time zone offset: [20/Dec/2012:13:39:40 -0800]
Request: "GET /index.html HTTP/1.1"
HTML Status: 200
Response size (bytes): 16844
Referrer: "http://google.com/com"
Time taken to server the request (seconds): 1

Also, don't include too many log parameters on web requests lest you suffer from scrolling blindness. The logs have to be readable, in real time, by a human. For unusual periods of peak traffic, it helps to tail and grep the logs in real time to focus on specific log entries.

You'll be amazed at how much you'll learn by simply monitoring these server logs on a daily basis, for just a couple weeks; it will give you a sense for what is normal traffic. Errors, unusual requests, and long response times should be investigated. Some of these issues will be web app bugs or areas requiring optimization which may have never been detected (such as an internal report taking minutes to respond when a typical request/response takes less than a second).

Frequent attacks go unnoticed until there is some ramification. If you don't know what normal behavior is then you won't be able to detect abnormal behavior.
 

What Are Objects?

In computer science, there is the concept of objects as it relates to object-oriented programming (OOP) which is sometimes confused with basic data.

For example, Amazon's S3 web service has a concept of buckets which hold objects. But, there's no reason why these objects can't be referred to as files since that's all they really are. Files are a collection of data on a file system. Data can be structured to represent many different things such as records in a database, images on a hard drive, or music on an MP3 player.

What makes an object special is that it is not only data, but has behavior, too. In order to see or act on the data inside an object, it must be accessed through its methods. Generally speaking, this is the only (and preferred) way of reading, writing, and updating an object's data.

This may not seem like a big deal, but it is since, whenever an object is passed around, all of its behavior is passed along, too.

For example, in a non-object world, I could read data from a database, manipulate it, perhaps improperly like incorrectly calculating tax or interest, and then update the database with the bad data.

However, if I pass along an object representing that same data from the database, the methods (i.e. business logic) that wrap and act on that data, protect the data's integrity. How the data is manipulated can be private, like a black box, but how you can access it is publicly described by its application programming interface (API).

I don't need to know how my car engine works or how gasoline is made, all I need to do is plug the gas hose into my gas tank. If I try to stick the diesel hose into the my tank, it won't fit.

The beauty of objects is that they can be plugged into each other and the data just flows. This is very different from previous programming paradigms, such as procedural languages, like BASIC, where each step of the program must be followed in sequence; or declarative languages like SQL where the programmers don't describe the steps to take, rather they declare what is to be accomplished, such as, "show me all the people with the last name equal to Smith."

Evolution of Computer Languages

It's interesting, yet not unexpected, to see how a next generation computer language becomes everything the previous generation wanted to be.

Java became what Ada wanted to be (write once, run anywhere), and JavaScript has become what Java applets wanted to be (web browser executable code).

Apple Languages

Over the last 15 years, or so, Apple has tried new languages while never giving up their tried and true Objective-C. One nice thing about Objective-C, which eased its learning curve, is that it's a superset of ANSI C. Most any ANSI C code will run in Objective-C.

Apple's current Cocoa API goes back to the birth of NeXT in the late 1980s. In the early days, app kit and foundation kit, which carried over into Cocoa, were the bees knees. If you didn't code on NeXT technology before 1995, then you probably didn't write object oriented code, commercially, until Java was released by Sun Microsystems. If you've ever coded in Objective-C, then you've probably figured out that all of those classes beginning with NS came from NeXTSTEP.

In the late 1990s, Java was too big for Apple to ignore, so they integrated it into WebObjects 3.5. Java and Objective-C are very similar. Java code is almost a one-to-one mapping to Objective-C and the two languages can be "bridged" such that code written in Java can be automatically and seamlessly converted to run in Objective-C and vice versa.

Trivia: The communication between Java and Objective-C is handled by a Java to Objective-C Bridging Specification, aka a .JOBS file. Get it?!?

War of the Languages

The best feature of Java is that it's a strongly (static) typed language.

The best feature of Objective-C is that it's a weakly (dynamically) typed language.

This type of thinking can lead to holy wars between coders as they argued which language was better. To avoid sending mixed messages, Apple made it clear about a dozen years ago that Java was Apple's server language of choice, while desktop apps would be developed in Objective-C.

Apple did experiment with Java desktop apps when Mac OS X was released ten years ago by integrating Java into the operating system. But Java, just like Flash, isn't a good desktop environment for many reasons. If memory serves, I believe Steve Jobs referred to Flash as a CPU hog and Java as a pig. Both Java and Flash run inside virtual machines which is a completely isolated operating system within your own operating system. It's literally the equivalent of speaking to someone else through a translator. The virtual machine must first be loaded and then translate the Java or Flash code into machine language at run time. Compare that to Objective-C which can be compiled directly into machine code that runs natively.

It seems strange that a language developed almost 30 years ago is still cutting edge, today, until you realize that Unix has been around since 1969.

Tricks I Learned At Apple: Steve Jobs Load Testing

When I worked at the Apple Online Store, we would never load test the live website. There was rarely a need. However, it was always an interesting experience to turn the store back on after Steve Jobs walked off stage following one of his keynote presentations. As part of our postmortem, once the store was back online, we'd ask ourselves where the servers were constrained: CPU, network bandwidth, disk I/O, or memory? While it's hard to predict exactly how the entire system would behave in the real world, we had a good idea, before we flipped the switch, thanks to our thorough testing strategies.

Load Testing
Many companies use load testing software to see what kind of load their web app can handle. A common, but flawed way to load test a web site is to bring it online and then turn on the load testers. The problem with this technique is that it doesn't give you any idea of how the website will perform if it goes down while it's live. When a website, that is in production, goes down, it must be brought up while under load and things will behave very, very differently. For example, it was discovered, when the iTunes Store first launched, that one of its trusted WebObjects components wasn't thread safe and this bug only presented itself under very heavy load.

Cutting My Teeth
When I first joined the Apple online store, I was paired up with an experienced software engineer so that I could get up to speed on the code repository, build process, and unit and component testing. Since the online store was already live, we would never roll out new code without first testing it and gathering detail metrics.

My first task with my coworker was to implement a simple web service which retrieved product information, in the form of a plist, over the network. A simple service like this could normally be written in a day or two, but it took us most of the week as my mentor explained each step to me while we pair programmed our way through the process. (Although we pair programmed, our software methodology was Agile/Scrum, not Extreme Programing. Each team used whichever development technique they agreed on as long as they could stay on schedule. The teams I worked with were fortunate to have formally trained scrum masters who were supported by management.)

Before writing any production code, we'd write our unit tests. All software engineers should be taught to write their API unit tests first – it's a good discipline to learn. Next, we coded using WebObjects/Java with Eclipse/WOLips and we always ran the app in debug mode with key break points so that we could step through the code. I've frequently seen too many software engineers, elsewhere, who just code away as if they're throwing something against the wall to see what sticks.

As soon as we checked in our code, the repository would automatically build all of the applications and run the unit tests against them. If you broke the build everyone on the team, plus a project manager or two, would receive a notification e-mail identifying you as the culprit.

Token

We had one, highly specialized piece of software code which could only be checked out, worked on, and checked in by a single engineer at a time. You were only allowed to touch this piece of code if you possessed a physical token. In our case, the token was a Darth Tater doll, which had to be conspicuously displayed on the top of your cube or bookcase.

Gathering Metrics

Once our service was code complete, bug free, and checked into the repository we began component testing to gather metrics on the new code. This is another step that's commonly overlooked in novice teams. I suspect that this "gather metrics" step isn't included in The Joel Test because Joel Spolsky's product was a desktop app and not a web app under heavy load (or, perhaps it's implicit in Spolsky's "Do you have testers?" step).

Before we could even consider including our code in the live code branch, we would hit it with many millions of requests. At Apple, we had very sophisticated caching algorithms which could store any number of entries we wanted, depending on our goals. Did we need a cache with only 500 products in it or 50,000? After a cold start, would we need to "warm up" the cache with specific products? How long should we wait, after no hits, before removing a product from the cache to free up memory?

As a side note, our caches were always hash tables. The beauty of a hash table is that it has a Big O notation run time that's constant: O(1). When you're asked, during a job interview, which is the fasted lookup function, don't, as is very common, say, "a binary tree." Perfect hash tables always win, hands down.

Tweaking and Done

We would tweak our code until we had acceptable metrics. Our metrics would measure how much memory the cache used and how long it took for each service request/response to be fulfilled. Depending on our needs, we might shoot for a goal to have 99.7% of our service requests returned within 35 ms, while 95% were returned within 10 ms with no single request taking longer than 50 ms.

These tests were run against a copy of the live database in a production environment. It's not a perfect indication of how the web app would perform once it was live. But it doesn't take long for this to be a great way to set expectations.

At the end of our sprint these metrics would be demoed as part of the Agile definition of "done." The code was now ready to be checked into the QA branch for functional testing before going live.

Push Notifications on the Cheap

One of the problems with feed updates, such as an RSS or a podcast feed, is that they're not real time. Each client, that subscribes to a feed, will continuously poll the server for updates (usually every thirty minutes). The two leading solutions to solve this problem are rssCloud and PuSH.

Real time notification can be somewhat of a challenge due to the fact that most clients are behind a firewall. Punching a hole through a firewall is too much to ask of the casual user, plus it could potentially open the client to security issues.

Even for a client that has solved the problem of negotiating the client's firewall, there's still a bandwidth issue. Each push notification server must maintain a connection with its subscribed clients so that it can notify them of updates; this is usually done via a socket which is a constantly-open IP connection. The problem with this is that there's a bandwidth cost to keep this connection open.

Push Without The Bandwidth Overhead Costs

A workaround that I'm visualizing can avoid any unnecessary bandwidth push notification costs charged to the publisher/aggregator by simply using Amazon's Simple Emil Service (SES). Here's how it would work:

1. The client app would be configured to check a client's e-mail address for notifications.

2. Notifications would be sent to subscribers using SES.

3. When the client received the notification e-mail, it would then ping the publisher to receive the update.

The cost benefit of this is realized by shifting the notification bandwidth channel from the client and publisher notification server to the client and their mail server. Most data centers will charge their customers based on bandwidth, so there's a cost for a publisher to maintain a push notification channel with all of their clients. But, there is no cost for a client to maintain a connection with their mail server. In the past, new e-mail notification relied on client polling (usually once per minute). However, most modern mail clients now rely on the IMAP IDLE feature for immediate notification. New notifications would be received within a few seconds by having the subscriber's client implementing IMAP IDLE.

The e-mail account notification could be implemented two different ways. The publisher could maintain their own e-mail account which all of the clients check or, each client could maintain their own e-mail account. The former option would only work with a small number of clients. The latter option allows the publisher to manage traffic by pushing out notifications over a short timeframe to manage the "thundering herd" problem if scaling is an issue.

While this is not the most elegant solution, it eliminates all unnecessary bandwidth overhead costs involved with maintaining a notification channel. The only costs incurred by the publisher are for bandwidth used to publish the notifications and updates.

Beating Back Facebook Sharer.php Bugs

Epics3.com is a fairly simple site for sharing your photos on the web. It ties in nicely with Facebook and Twitter by either using simple share buttons at the bottom of each large image or through an API when the photos are uploaded via a browser or through e-mail.

On Twitter, the photo's caption is tweeted out with a link to the full size image. In the case of Facebook, though, just the caption is sent over using Facebook's API and it's posted on your wall as a link. Facebook had documented some hints which you can put into the web page's HTML so it can pick up a specific thumbnail of the image to post on your Facebook wall.

Bugs
Unfortunately, there is a Facebook bug, that pops up randomly, where the hints for finding the thumbnail image are ignored. This results in the first image on the webpage being posted to your wall. In the case of Epics3, as in most, this will result in the website's logo posted on your wall instead of the photo. Bad, bad, user experience. How could it work for months without a problem until recently?


click to enlarge

Documentation
Facebook's API documentation is notoriously horrible (on the other hand, Amazon Web Services documentation is the textbook case on how APIs should be documented).

After spending several hours tracking down this bug on the Internet, I concluded that no one had solved it. (I also doubt that it was well documented in a way that Facebook, Inc. can reproduce the problem). When searching for the bug I found many cases of users who had encountered the same problem with shrugged shoulders. After all, this is just HTML - there's no logic to debug.

Details
When you use Facebook's sharer.php button to post a link to your wall, Facebook's servers will take a quick look at the link and spider the webpage to find a suitable photo. You can tell Facebook exactly which photo to post next to the link by adding this HTML code to the HEAD of your webpage:

<link rel="image_src" href="http://example.com/thumbnail.jpg"/>


For some reason, this hint has worked intermittently, lately - most notably it has been failing on the weekends.

User Agent Solution
After several hours of no joy I finally gave up trying to get it to work according to Facebook's API. To solve my problem, I placed a conditional surrounding the entire web page's body. When an incoming request's user agent contains the word facebook, I return a webpage with nothing other than the image. [The full Facebook user agent is "facebook share (http://facebook.com/sharer.php)", but just looking for the word facebook should be enough.] When the user agent in the request is anything other than facebook, I simply return the normal Epics3 webpage.

Problem solved (I hope).