Planet Web 2.0

The Adobe AIR platform is one of the hottest desktop platforms available. While we've given you numerous recommendations on great AIR apps, finding them all in one spot (outside of ReadWriteWeb) can be a pain.

With so many to discover, here's a look at why you might want to recommend your favorite Adobe AIR apps at FreshAIRApps.

FreshAIRApps officially launched in April to showcase some of the most popular AIR applications available. James Whittaker, the interactive designer and creator of FreshAIRApps, does a great job of taking FreshAIRApps a step further than your typical directory. FreshAIRApps intends to give back to the community by providing users and developers with Adobe AIR news updates, tutorials, and numerous development resources.

FreshAIRApps Features

Adobe AIR Showcase
FreshAIRApps provides a great showcase for some of the hottest AIR apps available. Given a brief review, users are welcome to provide their own opinions in the comments section and rate each app on a scale of 1-5. So far only a handful of of AIR apps have been submitted. However, site visitors can easily submit an AIR app for reviewing via the "Suggest an AIR app" button located conveniently on the site's navigation menu.

Adobe AIR News
FreshAIRApps also has a section devoted to Adobe AIR news. What's really unique about this section is the type of news that's featured. Not only will visitors receive updates on the latest news from Adobe, but also news about applications that have been reviewed including upcoming releases and links to interviews of the developers.

Adobe AIR Tutorials and Resources
Due to FreshAIRApps' recent debut there are only two tutorials available. The provided tutorials are aimed at beginning developers of the Adobe AIR platform. The first tutorial features a host of information about the tools and technical skills you'll need to begin developing an application using the Adobe AIR platform. Also provided are links to Adobe AIR development extensions for Dreamweaver CS3 and Flash CS3.

The second tutorial provides developers with backgrounds in programming languages such as Flex, Flash, HTML, and AJAX with a great list of documentation resources to get started. Along with book recommendations, more resources can be found in the 'Resources' section of the site.

Supporting the Adobe AIR Community

Because Twitter is getting more popular, every glitch in the service is now felt more acutely. Going without Twitter for many people is even harder than going without email, and so outages lead to complaints. Complaints pile up and become debates, asking questions like: should Twitter be converted into a protocol and become decentralized? Is that the way to scale Twitter and make it more reliable? If not, how can that goal be accomplished?

To me, the answer to decentralization is a firm no. First of all technically it won't solve the problem. At least not in any way that Twitter folks can't solve it themselves. The whole question actually misses the main point. We love Twitter as an application, and its strength is the fact that people know where to find it, people love the What are you doing now? question. Amongst a sea of copycats, it was Twitter that took off and that's why we know and love it. Twitter is Twitter and it should not be anything else.

The question that people should be asking, though, is how to properly scale Twitter and, for that matter, the whole slew of other life streaming applications. Clearly these applications are trying to break new ground by merging streams of your friends' activity together and presenting you with a single view of that information. All of these applications are facing similar challenges and they could all architecturally benefit from the same pattern. So what is it? How should these applications be designed so that they scale to meet the demand?

What is the Problem?

To understand the challenge facing Twitter and other life streaming applications look at the diagram below. A node in the center is an individual user and the the nodes around it are the users that follow or subscribe to this user. The blue nodes are subscribers and red ones, U1 and U2, are publishers in this diagram. The yellow node, R1, is a receiver that gets updates from both U1 and U2. The yellow note shows that the stream of R1 contains updates from both U1 and U2.

How to manage all of this is pretty obvious until one starts thinking about how to generate the stream for R1. The answer that comes to mind right away is - on demand. That is, when the user R1 checks his Twitter page or via one of the many Twitter clients, the stream is computed and delivered on the fly. Unfortunately, and this is true with distributed large-scale systems in general, the first answer is likely to be wrong. If the user subscribes to say 100 other people, to pull together 100 streams and merge them so that they are in the right time order is likely to take seconds if not minutes. Who is going to want to wait this long?

If the user view can not be computed on the fly, then the only other answer is that it is pre-generated. That is, whether the user checks or does not check the stream, it is there and available as soon as possible. This answer also seems wrong, because this approach is quite wasteful. Pre-generating all these views without anyone looking at them is going to cost a lot of wasted space and compute power. Yet, of the two approaches, this is the one that delivers a better user experience and so this is the one that is likely to succeed.

Relational Database - Likely the Wrong Way

We wrote about scalability challenges with relational databases in our post about Amazon Dynamo. Yet relational databases should not be dismissed. First, lets look at how relational database can be used to engineer a solution. A simple approach is to have three tables USER, FRIENDS, and MESSAGES; the tables would look like this:

This is very simple, and of course it does not scale. As the tables grow, even if everything is indexed, doing all the look ups via this set of tables can not work. What kills this solution is that all information of all users is sitting in these tables. What if there was a way to split all this information into many databases? If this was possible then the system would scale. So lets say we took all the users that start with letter 'a' and put them into one database and then all the users that start with letter 'b' into another and so on. There are systems that can benefit from such partitioning. For example, MyBlogLog can be partitioned so that all data is stored around individual blogs.

Unfortunately, in case of Twitter and other life streaming applications such partitioning does not work. The reason is that we can not predict who is going to follow who. So there is no way to cluster the data so that all the necessary messages end up in the same database. And cross-database queries are very costly - not a way to go. So the reason a relational database does not work is because a single database system does not scale and the data is not amicable to partitioning into multiple databases.

Data and Compute Cloud - Possibly the Right Way

Increasingly more and more companies, particularly in the consumer space, are turning to cloud computing. The fundamental difference that the cloud approach takes vs. the relational database approach, is that with the cloud you split the data and split the computation on a massive grid and then make trade-offs. For example, in the relational database data duplication is a big no. Things need to be normalized. In the cloud world data duplication is okay, because a lot of views are pre-generated and a lot of them contain duplicate information. For example, a message from one user is duplicated into bins for everyone who subscribes.

The second trade-off is consistency, which is kind of related to duplication. When we are talking about mission critical applications on Wall Street, data consistency is a top priority at all times. But does it matter if user A gets a message from user B and then it takes a few minutes to deliver the same message to user C? Of course not. Even in our real-time hungry culture this delay is acceptable. So the trade-off is to forgo consistency across the entire system but gain flexibility instead.

The cloud version of the system is depicted above. For simplicity we have 26 machines on the grid - one for each letter. Each machine is identical and focused on users whose id starts with the corresponding letter. All information about a specific user is stored on the machine, including profile, list of people who follow this user, etc. The key thing is that the system maintains not only a list of messages generated by this user, it also immediately creates a list of messages for the user. It does that by processing updates from other machines. The whole thing is a peer-to-peer, completely connected ring system, where each machine is connected to all others.

When a user types an update into the UI it is first sent to the machine that handles this user. That machine immediately updates the user's messages and then broadcasts those messages to other machines. For each other user who follows the one that just updated, the system constructs the message and sends it precisely to the machine that hosts that other user (it is known because users are split based on letters). So for each update the system sends out exactly the correct number of messages. Then as soon as the message is received on the other end, it is written directly into the stream of the receiving user.

What is nice about the cloud solution is that it scales as the user set grows. The partitioning mechanism does not need to be based on letters, in fact it should really be based on a quick, uniform hash which would ensure that the users are distributed evenly around the grid. And there are many other details, of course, because what we described here is not the solution, but just the basic idea.

Conclusion

Software engineering as a discipline has evolved a set of design patterns. These are solutions that work across languages and across different systems. What we need is new design patterns for life streaming applications like Twitter. Instead of talking about breaking the service into distributed systems (which likely would perform worse and have more outages), we need to work through the best ways of building this kind of software. Google scales, Amazon scales, and as Stowe Boyd pointed out, so does AIM (although it has considerably less connections). And certainly there is a right architecture for Twitter, which would make it scale to meet the growing demand.

So lets start right now! Please tell us what you think are the best ways to design life streaming applications? What trade-offs do you think should be made? And please tell us about your specific experiences building these systems.

Yahoo! today released a developer preview of its Yahoo! Internet Location Platform, a collection of in-depth geo-location based APIs. We expect to see location be more smartly used in many applications around the web thanks to this platform.

The gist of what's being enabled is this: applications can provide the name of one location and then the Yahoo! APIs will report neighboring and "parent" locations. Flickr developer and map lover Dan Catt articulates the potential power of the API very well in a blog post today.

A lot of Ground Covered

Yahoo! explains the breadth and depth of location data it now offers thusly: "The [Platform] contains about six million places. Coverage varies from country-to-country but globally includes several hundred thousand unique administrative areas with half a million variant names; several thousand historical administrative areas; over two million unique settlements and suburbs, and two-and-a-half million unique postcode points covering about 150 countries, plus a significant number of points of interest, Colloquial Regions, Area Codes, Time Zones, and Islands."

Geolocation is Hot Everywhere

Geolocation is hot, a number of new projects are underway to leverage increasingly sophisticated geographic knowledge to deliver value to end users. See our coverage of Brightkite and of Yahoo!'s own excellent FireEagle, for example.

Flickr developer Catt explains, for example, that Flickr could use the new APIs to offer images of nearby photos on several different levels, with accuracy as granular as Flickr is able to output.

There are a lot of interesting possibilities, not just for mapping but for services that are map aware. What would you like to see turned geo-smart? We're excited to see what developers come up with. We probably won't have to wait for long, either, since the Platform was released the day before O'Reilly's Where 2.0 conference begins in Burlingame, California. Keep your eyes peeled for location savvy apps this week!

As our network blog AltSearchEngines reported this morning, the long-awaited and much hyped natural language processing search engine Powerset launched this morning. Kind of. For now, the search service only uses Wikipedia and Freebase as source material for answers to your query. So it's not really fair to compare it to Google yet, but this is a search engine, and that means it will always be held to the gold standard set by the market leader.

Comparing the two is tricky, since Google searches the entire web and Powerset only processes two sites. The admittedly not very scientific method that we came up with was to compare a handful of searches on Powerset, to the results for the same query on Google restricted to "site:wikipedia.org."

Powerset does some interesting things with general queries, such as displaying "Factz," which is an ontology showing various concepts related to your query and how they relate to one another, or "Dossiers," which are a summary of key information about your query. Sometimes it yields some odd results (such as this query for "ants" for which the key finding is that ants are "a fictional race from the video game Crash Twinsanity.") However, the real promise of NLP search engines, in our opinion, is that users will be able to make search queries using natural language -- or in other words, by asking a question. So we chose a few questions at random -- things we knew Wikipedia would have answers for -- and threw them at both Powerset and Google.

Query: Who invented dental floss?

Powerset's answer for this query was curious. The number one result comes from the Wikipedia entry for dental floss and highlights this line: "It was around this time, however, that Dr. Charles C. Bass developed nylon floss." Charles Bass, however, is not the correct answer. Earlier in the same article is this line, "Levi Spear Parmly, a dentist from New Orleans, is credited with inventing the first form of dental floss." Why didn't Powerset find it? It's second results, which comes from a Wikipedia entry on scientific achievements from the year 1815, correctly highlights Parmly as the inventor.

Google performed poorly for this query. The same 1815 article is identified in the sixth spot on the results, with the sentence mentioning Levi Spear Parmly highlighted, but the first few results aren't even close. Even though that's not as impressive as Powerset's results, both would require a user to click through to the article to verify the answer (because Powerset returned two different answers), and is scrolling to the 6th spot really that taxing? Taxing enough to make you switch to a new search engine? Interestingly, this query set loose on all of Google does quite well, returning the correct answer in a link to a trivia site in the first result.

Query: What is the capital of France?

Not surprisingly, both Google and Powerset nail this one. Both point to the Wikipedia entry on Paris, France in the number one spot with the sentence, "Paris is the capital of France" highlighted.

Query: Where is Paris?

This is a fundamentally more challenging query, because there are a large number of cities and towns called "Paris" in the world. And not surprisingly, neither search engine gives what we would call a "perfect" result.

Both return the article on Paris, France first. On Google, that's followed but a handful of other articles about the city and one about Paris, Tennessee. On Powerset, the second article is about Paris Hilton -- um? -- followed by one about Paris, Texas, and in fourth place the most helpful article it could have returned, the disambiguation page on Wikipedia for Paris. (Oddly, with the question mark, the query returned "Paris, Missouri" from Freebase, and without the question mark it returned "Paris, Texas.")

On Google at large, the results focus almost exclusively on Paris, France.

It would seem that both search engines generally understand that "where is Paris" means that Paris is a place (though upon reflection, perhaps we could have been searching for the location of Paris Hilton...), but neither recognize very well that it could mean any number of different places.

Query: Who is Joey Tribbiani?

Both Powerset and Google correctly call up the article about this fictional character in their first spot, but Google actually does a better job of highlighting who he is. Compare:

• Google: After the 2003/2004 final season of Friends, Joey Tribbiani became the main character of Joey, a spin-off TV series, where he moved to L.A. to polish his ...
• Powerset: In the end of the series, Joey was the only Friend that ended up without a lover or a spouse, even though he is the one that dated the most women. ... Joey becomes good friends with an attractive female attorney named Alex, who, along with her husband, a travelling [sic] musician named Eric, is Joey's landlord.

Google has the name of both shows in which the character appears in their excerpt, while Powerset's excerpt is made up of information about the series' that only someone who already knew the character would understand (without clicking through to read the full article) -- and it doesn't differentiate between the two -- before the ellipses the excerpt is talking about "Friends" and after it is talk about "Joey."

Google at large also finds the Wikipedia article first with the same excerpt -- it also finds clips of the show on YouTube, and the actor's (Matt LeBlanc) IMDB entry, as well the official site for the spin-off "Joey."

Conclusion

This was really just a very quick and informal test, and we barely put Powerset through its paces. But our first snap impressions are that Powerset doesn't do a markedly better job of finding answers than Google for most queries. Some might argue that we didn't play to Powerset's strengths and frame our queries properly, or search for things obscure enough to notice any differentiation. But the promise of natural language search is that people don't have to learn how to search -- they can just ask questions as they normally would. We also can't expect that everything they're going to look for will be obscure and hard to find via traditional search engines -- more often than not, they probably won't be.

Powerset will have an immense uphill battle to make any sort of dent in the search market. Google controls 67% of searches in the US, and the top 4 search engines make up about 98% of searches. If Google remains "good enough," Powerset will have a hard time convincing people to switch. It will be easier to make a judgment about the company's future as a real Google competitor once it is crawling more than two sites, however.

What do you think about Powerset? Impressed? Not impressed? Let us know in the comments below.

Marketing consultant and web connoisseur Noah Brier has launched a simple but fascinating project called BrandTags.net. The idea is that visitors are shown a logo, we respond with a word or very short phrase that we associate with the corresponding brand and then we're given the option to view all the "tags" given a brand in a big tag cloud.

It's a simple but elegant and interesting experiment. The tag cloud for Walmart, for example, shows that the word "evil" is pretty big - but "cheap" is even bigger! We've embedded the site below in an iframe if you want to try it out yourself.

Nice Touches

One of the nicest touches here is how Brier displays the tags in oversized font. By requiring users to scroll down the page, we get to enjoy thinking to ourselves "surely this is the largest tag for this brand" - only to scroll on and find that another term is even more frequently associated with that company!

One thing that would be nice would be to have comments be enabled at the bottom of the tag cloud screens. That way people could explain to those who don't know why, for example, the word "racist" is so large on Tommy Hilfiger's page.

BrandTags may not be the kind of site that consumers regularly return to, but it's fun to try out once. Obviously it's something that companies would have a real interest in checking out, especially if it takes off. Brier reports that it's recieved over 77,000 tags in the first weekend it was live.

We've got it in an iframe below, just because if iframes are good enough for Google Friend Connect then gosh darn it, they're good enough for us too. Click through some brands on there...you just might find ours and get to offer a little feedback!

If you're reading this post by RSS you can click through to see the iframe or visit the full BrandTags.net site itself.

Later tonight Google will launch a new service called Friend Connect, aiming to "bring the social" to any page around the web. Unfortunately the service takes a bunch of open technical standards yearning to see the light of day through mass adoption and puts them in a dark little box where they will struggle to breathe.

Google could have worked with other large companies and with the creators of these standards (some are in the Data Portability Working Group that Google joined, for example) to tackle the hard questions around data exposure, integration and privacy. Instead they are pushing their Open Social standard around in an iframe. Easy is very good, but co-operation could have come up with something better than this.

The Plan

Friend Connect uses OpenID, oAuth and Open Social to let users log in to their favorite apps using a trusted ID provider and then access their friend info from those apps - all while on another website altogether. That could be any website that has chosen apps from the Open Social/ Friend Connect app gallery and pasted the iframe code onto their page.

The embrace of these standards is great news - but the implementation is wanting.

If you'd like to hear the press call, we've got it up on our podcast site ReadWriteTalk, or you can listen to it in this wonderful iframe below (feel the social!). My questions come in at around 19:00, press the "play now" link to listen. The first part is boring and scripted but the Q&A part is pretty fun.

Who's In?

Below is a splash page to-be for the Friend Connect service - you'll notice that Facebook friends are included in the discussion. Facebook isn't a participant in Open Social though, so we're not clear on how they are a part of this Open Social In a Box Initiative. Google says it's using the publicly available Facebook API. Google says that its Social Graph API, which indexes friends everywhere and that we called a creeping privacy violation - is not included in Friend Connect simply because they didn't need it. It's unclear on what information site owners will be able to put in their iframes.

You Can't Use it Yet

While the whole developer and publisher world is anxiously awaiting details from the launch tonight, Google is putting a damper on adoption by limiting the Friend Connect "preview release" to a handful of white listed apps and a short list of selected websites. The company says it has to prove it can scale the infrastructure (ooh, can Google scale? I don't know, better limit the approved users to just a tiny handful!) and it wants to see what kinds of features developers and site owners want to request. Apparently the company believes this feedback is best done by making said parties look from the outside and send emails guessing about what they'd like to see once they are let inside. This seems completely backwards to me.

You Can't Touch What's Inside the Magic Box

Site owners will be able to add Open Social apps to their web pages - sort of. They'll be able to display them inside an iframe, a separate web page inside a webpage. They won't be able to leverage that user data to change what they deliver themselves to their users.

Apps in an iframe may as well be a social sidebar ala Flock or Yoono. Those collections of social apps are probably more useful anyway.

Conversations Are Complicated

Google made it clear during their press call that they are aiming for the easiest, simplest and safest way to enable social apps to be integrated into other websites. It will take less than six months, they promise.

Let's be clear that it's not going to be easy to figure out how to enable all this user data to be mashed up in acceptably safe ways. We asked Google how they can assume that one user's friends on IMeem have permission to access their info out on other sites around the web. They said that users will have to be given the option whether to expose that info to third party sites or not, something we haven't seen any details on yet from the original source social networks. That would be even more difficult if the destination sites had read, much less write, access to that ported-in social networking data.

Everyone Is Talking, Though, Right?

Those hard questions are the ones that these companies are supposed to be working on together through the Data Portability Working Group, though. The Group has published best practices documents tackling a number of difficult questions already. Today when asked why all these companies were making separate announcements, though, Google said that the beauty of open standards is that companies don't have to talk. They can just meet up around interoperable technologies.

We're hoping that the biggest social companies in the world are talking, though. We'd like to see them complete and advance these standards, not just implement them by themselves, in a little box.

The Data Portability Working Group is a challenging situation and it may not be a surprise that Google and the other large players aren't spending as much time there as they are on engineering. The end results of the engineering though, are in this case and in the case of MySpace's announcement earlier this week, appear a fair cry short of the dreams that fuel the data portability community.

We'd like to see the social truly opened up and for all the vendors and consultants, large and small, to work together to get through the permissioning and safety questions that ought be be just small bumps on the road to a universe of innovation.