Are you building a large scale consumer web service? Below are some helpful talks from companies that have done so successfully, using approaches including memory caching, replication, feature based sharding, and horizontal (user) sharding.

Nat Brown, iLike’s CTO, spoke at the Google I/O conference on scaling and analytics techniques , with a particular focus on OpenSocial.

Saran Chari, CTO and co-founder of Flixster, has an overview of the data scaling techniques used at Flixster

I love the guiding principle of “Spirit of the Game” from the Official Rules of Ultimate.

Politics and indeed life would be better if this principle were more widely applied.

Just about everyone knows this already, but I’ll make the official announcement:

I am very pleased to be joining iLike as an engineer.

Some of the things I like about iLike: the team combines smarts with personal integrity and minimal ego; they ship every week and iterate and learn rapidly; they have a small team and flat org structure without excessive overhead for meetings, process, or politics; they’ve built a valuable service with tens of millions of satisfied users (their FB app ranks among the top in terms of user ratings); each developer owns substantial feature areas; and they have interesting problems in scale, data mining, syndication, and user experience. (Several fine developers I’ve worked with in the past are here, including Mark Aiken, Scott Haug, and Ray Fortna.)

iLike has achieved a truly impressive amount of traction in a short time with a small team. They are promoted on the back of every Ticketmaster ticket and have been repeatedly cited by Mark Zuckerberg as an example of a valuable Facebook app.

I can’t talk about the first thing I’m working on yet but will post details here when it ships.

Update: Use ack instead. It’s great.

grep-find is an Emacs function for recursively searching all of the files in the current directory. By default, it recurses into Subversion directories, producing redundant, spurious hits.

Fortunately, it’s easy to fix this. If you add the following lines to your .emacs, grep-find will have the desired behavior of ignoring Subversion’s copies of the base versions of your files.

(custom-set-variables
'(grep-find-command "find . -type f -not -name \"*.svn-base\" -and -not -name \"*.tmp\" -print0 | xargs -0 -e grep -n -s -F "))


For further simplicity, you can do something like the following to add a single-keystroke command for searching an entire project.

(defun grep-project (s)
(interactive "sSearch project for: ")
(grep-find (concat "find /home/philbo/webroot/ -type f -not -name \"*.svn-base\" -and -not -name \"*.tmp\" -and -not -name \"*.log\" -print0 | xargs -0 -e grep -n -s \
-F \"" s "\""))
)

(global-set-key (quote [f4]) (quote grep-project))


I am enjoying OSX and the Macbook Pro, but I was initially finding it difficult to train myself out of familiar Windows keyboard shortcuts like Control-X to cut the clipboard and Alt-D for the location bar in Firefox.

Here I’ll describe the steps I took to add Windows/Linux compatible shortcuts for the clipboard and Firefox to OSX. I also describe how to make Emacs in Terminal.app work better.

To add the the shortcuts, go to “System Preferences : Keyboard and Mouse : Keyboard Shortcuts”

Clipboard shortcuts:
Click the “+” button. Enter “Cut” the menu title and press “Control X” in the keyboard shortcut field. Do the same for “Paste” (Control V) and “Copy” (Control C).

Firefox shortcuts:
Click the “+” button and select “Firefox” for the list of apps. Add a shortcut for “Open Location…” (Alt D) and “New Tab” (Control T).

At this point you can use either the Mac or the Windows keyboard shortcuts and have the right thing happen.

Advanced options: Emacs

If you happen to be an Emacs user, you need to do one additional thing to make sure these shortcuts don’t interfere with Emacs. For example, “Control-V” should paste in every application except terminal, since that would break the Emacs “scroll-down” function.

Once again, click the “+” button and select “Terminal” under application. Enter “Cut” the menu title and press “Command X” in the keyboard shortcut field. This will disable the mapping of Control X in the terminal.

You’ll also want to take these additional steps to ensure that Terminal sends keys through to Emacs:

Under “Terminal : Preferences : Settings : Keyboard”, check “Use Option as Meta key”. Under Keys, remap the Home, End, Page Up, and Page DOwn keys to send these strings to the terminal.

Home = \033[1~
Insert = \033[2~
Remove = \033[3~
End = \033[4~
PageUp = \033[5~
PageDown = \033[6~

Many users have noted difficulties with GPS in Beyond411.

I’ve investigated and discovered that the web service I am using for reverse geocoding is incorrectly returning empty results in many cases.

I have notified the providers of the web service; I will post an update as available. I believe it should be possible to fix this on the server with no client changes.

I am getting really sick of watching this rerun for umpteenth time.

Once again, one party is offering tax cuts as an effective strategy to win elections, while the other party argues that those tax cuts will only benefit the wealthy.

If we can talk about race like adults, why can’t we discuss taxes like fiscally responsible adults?

When the government is consistently running a deficit, in both good times and bad, there’s no such thing as a tax cut.

Deficit spending disguised as a tax “cut” is just a loan from the government that we’re going to have to pay back with interest.

Once we realize that the offered tax cuts don’t really exist we can move forward on the debate.

The debates over class warfare and income distribution obscure this simple truth that all sides should be able to agree on.

We should come together on a consensus around simple fiscal responsibility. I am disappointed to see that McCain is backing away from his previous commitment to balanced budgets and that neither Obama nor Clinton seem to be able to introduce new thinking and language to the debate.

I found these comments by John Doerr on Green Tech interesting.

The notion that we need “reindustrialize” the planet matches my sense that the first industrial revolution was a hack that has no chance of continuing to scale or endure. Our next president must rearchitect markets so that “the right thing to do is the profitable thing to do, so that it becomes the probable thing to happen.”

“To get solutions that scale, we are going to have to find answers that are economic for all people everywhere. We are going have to use policy to harness innovation to make sure the right thing to do is the profitable thing to do, so that it becomes the probable thing to happen. There’s more money that flows through markets in a day than all the word’s governments in a year…

The energy market is $6 trillion. I like to say it’s the mother of all markets. Compared to that Internet, which is a big deal, this is much bigger, much more exciting. But the challenge is much larger. Going green–solving that problem will be largest transformation on the planet.”

Doerr said the entire planet needs to “reindustrialize” to adopt less-polluting forms of energy.

Many people have called for the equivalent of an Apollo Project or Manhattan Project in the United States to solve the energy challenge. But Doerr said that those, which were multibillion-dollar, single-government agency projects, “fail miserably to convey the size of the challenge.”

To underscore how little is being done at the federal level, he said government funding in U.S. research and development on renewable energy was less than $1 billion last year, while oil giant Exxon makes $1.1 billion in revenue a day…

He predicted that the three leading presidential candidates will address climate change regulation far more aggressively than the current Bush administration, which has opposed mandates and sought to stay outside United Nations-led climate talks.

Despite Doerr’s concern for inadequate action on clean energy, he touched on the question of an investment bubble in green tech. Overall, he said there isn’t a bubble, but he does see some problems.

“There’s too much money chasing too few good ventures, despite the size of this problem,” Doerr said.

From an interview of the Post’s Editor-in-Chief by MediaBistro:

We’ve really fallen behind the times here. We are in the process of designing a Web 3.0 generation website that we hope will leapfrog where the current news industry is, ideally, and create a reader news experience that has not been imitated in the industry.

We’ve hired the biggest name in the design profession: Roger Black. His work includes the New York Times, Bloomberg and Houston Chronicle. He has created a design that we’ve very excited about. … The goal is to go live in mid-May…

I’ve long believed that the era of newspapers determining what is fit to read, as the New York Times might say, is over and the key to web design going forward is allowing the reader to take control of their news experience.

The digital era allows us to take our readers along with us on the experience with news. Our design achieves that with the creation of inline video service to “news cubes” that allows people to flip in one direction and get different kinds of contact and flip another direction and get video, databases, interactives or ask a question of a field of experts.

I’ve long believed that news sites need to break themselves entirely from the vertical construction that the newspaper industry has relied on for years. … We’re doing a horizontalization of the news: If there’s a topic, a person, a location, or event, calendar event or news event that you’re interested in, you can horizontally slice the database of our news and find things. … It will also allow you to track whatever person or topic you want on a daily basis just by coming back to the theme page. … We’ll create 15-20,000 RSS feeds on narrow-casted themes. This will meet people’s specific news interests.

We’ll also create online communities, an area where people can socially network around the news. So people interested in unique topics or subjects will have this online community where they can share news and comment to each other. … We’ll have a ‘blogger in chief’ for each community. … They will most likely not be a Times staffer. .. We hope to launch 25-30 communities this summer.

The Subversion Merge Fairy and associated documentation is now hosted as a Google Code project at http://code.google.com/p/svn-merge-fairy/.

Please contact me if you are interested in write permissions to contribute changes.

(The Merge Fairy is a Python script that automates the process of merging changes from one Subversion branch to another, based on an XML configuration file that describes branches and their dependencies. For example, you might want bug fixes from the release branch to be automatically merged to the trunk branch.

In the event of a merge conflict or a build failure after merging, the Merge Fairy sends email requesting help from a human to make a manual merge, resuming automated merging once this done.)