Web development frameworks

Oct 26 2009   8:58PM GMT

The imposing heterogeneity of media applications

Posted by: Roger “Buzz” King
3D animation, 3D modeling, advanced Web apps, automating Web searches, continuous data, media applications, Multimedia, the Metadata Object Description Schema, Video, video containers, Web 3.0, Web development, Web development frameworks, XML Schema

This blog is dedicated to the discussion of emerging web technologies. Today, we look at a the rapidly growing world of media applications, and their impact on the Semantic Web.

The problem of searching for media assets.

We’ve already looked at advanced media, in particular video, audio, and animation data, in previous blog postings. In particular, we’ve looked at the subtle and complex nature of media asset semantics. We’ve seen that interpreting a piece of video, for example, is far, far more difficult than interpreting an integer or character field. Since the goal of the Semantic Web effort is to make the searching of the web highly automated, advanced media is becoming a huge and critical research and development focus for the builders of next-generation web development applications.

Just how do we provide an environment where media assets can be searched in a mostly automatic fashion, so that a human does not have to painfully paw through hundreds or thousands (or millions) of video chunks to find the right one? We’ve looked at emerging technologies for marking up advanced media information, and for making it usable in a variety of web applications. We’ve also looked at the dramatic challenge presented by mega apps to would-be users; the interfaces to these applications are truly massive and cannot present to the user the way in which they are meant to be used.

The problem of proprietary formats.

One specific, and very difficult problem, is the massive heterogeneity, not just of media formats, compression technologies, and container technologies, but of the applications themselves. If we are going to automate the searching of complex modeling, video, audio, and other media assets, we’re going to have to address a key question: since many media apps make use of their own proprietary data formats, how are we going to provide automated ways of searching media assets that are stored in these formats?

The problem of highly imperfect generic formats.

There are indeed many existing, as well as soon-to-emerge, standards for importing and exporting data between powerful media applications, but transformations in and out of these formats are often “lossy”, in that information is lost or changed. In fact, locating and downloading assets that are in supposedly-generic form is often very frustrating, because these assets end up not performing well. They can be difficult to edit and reuse. 3D animation models regularly blow up when animators try to import them into animation applications and the manipulate them. A hawk may look like a hawk until you try to render it with its wings flapping, and suddenly it’s a blob of geometric garbage.

One possible direction.

So, what do we do about the fact that many media assets must be manipulated by the original applications that created them? How can we facilitate reuse? It’s extremely unrealistic to expect users to master perhaps dozens of video or audio or animation applications. Filtering assets according to their file extensions is a good idea, and it is a well established practice.

But what we really need is a globally-known site that either literally or conceptually centralizes the massive network of import/export relationships, along with information about the relative success of these mappings. Are they ever lossy? If so, can they be fixed? What series of applications might we want an asset to be imported/exported through so that in the end it is in a usable format, given the applications that the user owns and has mastered?

There is much to be done. Right now, searching for and reusing media assets is a painstaking, trial-and-error-prone process.

Oct 11 2009   11:07PM GMT

Making information management scale: leveraging metadata on the new Web

Posted by: Roger “Buzz” King
3D modeling, automating Web searches, databases, DB2, information, Multimedia, MySQL, Oracle, PostgreSQL, RDF, Semantic Web, Video, Web 3.0, Web development frameworks, Web3.0

Previous postings of this blog.

This blog is dedicated to advanced Web development tools and concepts. Previous blog postings have focused on the emerging Semantic Web, which promises to make the Web radically easier to search and to greatly enhance the value of the vast sea of currently-disconnected information spread across the Web. We have also looked at Web 3.0 efforts, which promise to make multimedia websites highly usable and capable of conveying far more information than the current generation of websites. Previous postings describe breadth and depth of cutting edge Web technology.

Metadata: making that ratio small.

Here’s something that’s very important: Much of the ongoing research and development that is loosely categorized as Semantic Web and Web 3.0 efforts is focused on a specific technical goal, one that has been at the core of information management technology since the mainframe era that was epitomized by the IBM 360 series. That goal is to leverage metadata as much as possible.

It’s our best weapon against the truly staggering amount of information on the Web. This includes traditional text-based and numeric data, as well as books, medical advice, photographs, entertainment and training videos, music and recorded books, investment information, educational materials, scientific materials, e-government information, etc., etc. How can we possibly organize information and then search it in a way that scales? The Web is far from a closed world. In traditional data processing environments like banking, insurance, and credit card processing, we could get our arms around all of the data, as vast as it may have seemed. But the world of information today is an open world, effectively infinite in size.

Very informally, if you look at the size of the metadata divided by the size of the data itself, the smaller that fraction the better. In traditional relational databases (built with database management systems, such as Oracle, MS SQL Server, MySQL, PostgreSQL, or DB2), the extreme focus on minimizing this ratio has enabled the fast processing of extremely large volumes of data. The tradeoff is that the table definitions (or the “schema”), which form the heart of the metadata are very, very simplistic.

The old days: relational database schemas.

An insurance claim may be defined as a table with such columns as Subscriber_Name, Medical_Provider, etc., and thus, may consist of little or no more than a series of simple character and numeric fields. But if we need to process fifty thousand of them tonight, we must be able to bring many such table rows into memory at once, and quickly move through them. The database world was an extension of the paper world: a row in an insurance claim table was effectively an electronic successor to the traditional claim form.

Today: a far more challenging problem.

But on the new Web, information can be far more complex in nature, making the metadata to data ratio far larger. We’ve looked at some of the emerging technology and technical trends for embedding metadata in advanced forms of data (and for processing that metadata); this data includes books, images, video, modeling and animation, and sound. This new generation of information formats make up our personal health records and medical records images, industrial training materials, university “distance” courses, and the like. Each instance of these tends to be far more unique than individual insurance claim forms. And, it takes a lot of metadata to properly convey their “meaning”.

The challenge.

What we’re struggling with right now is to succinctly specify the meaning of modern media assets and to automate searching based on this metadata. This is our only hope for leveraging that ratio of metadata size divided by data size.

Oct 3 2009   9:12PM GMT

Multimedia: The Problem of Subtle Semantics

Posted by: Roger “Buzz” King
3D animation, 3D modeling, advanced Web apps, automating Web searches, blob data, continuous data, databases, information, Multimedia, rich internet apps, Semantic Web, smart search engines, tagging, Text, Web 2.0, Web 3.0, web applications, Web development, Web development frameworks, XML

The challenge of the Semantic Web.

We’ve looked at the emerging Semantic Web technology in the previous postings of this blog. The idea is to have a far, far smarter Web, one where the process of finding and interpreting and making use of far flung information can be largely automated. This is in sharp contrast with today’s Web, where these things have to be done in a painful, extremely time-consuming fashion.

So that is the key challenge. It has to do with searching the kinds of information that are important to us in our daily lives. This information, as it turns out, is very difficult to process automatically. Why is this?

The complexity of modern multimedia.

I teach a very basic 3D animation class to mostly computer science students. We use Maya, arguably the most popular 3D animation application, one that is used in the making of many animated features. The interesting thing about animation is that it is truly multimedia. It can give us a lot of insight into what we need the new Web to do for us.

That’s because the number and diversity of applications that are used for drawing, documenting, modeling, animating, motion capture, texturing, video rendering, video editing, video conversion and compression, sound editing, in even small projects, can be very impressive. Correspondingly, the wide variety and complexity of media formats involved in an animation project can be overwhelming.

What happens in an animation project? The workflow might begin with vector storyboard drawings to break the story down into scenes. In a typical animation project, 3D models in a variety of proprietary formats are used. Models must be transformed as they are exported from one application and imported into the next. Multiple video renders of animated models are made, and they must be edited together, along with multiple sound files. Multiple video and audio formats might be used. 2D images are used for textures; photographs of butterfly wings can be used to make an animated butterfly very realistic, and a checkerboard image made with Photoshop can be used to make a Linoleum floor. And along the way, a variety of note taking, screen capture, and conferencing software might be used to facilitate group communication.

There is also a heavy focus on reuse in an animation project. Building every model, editing every texture, creating every environment and background, recording every sound from scratch is frequently intractable. If existing assets cannot be tailored and reused, the project would be far too expensive and time consuming, and would demand too wide a variety of professionals to always be available. This raises the multimedia stakes, as assets of widely differing forms must be constantly reconfigured and used in concert in new ways.

But what’s the real problem? We aren’t all trying to produce complex animated videos. But very interestingly, in our everyday lives we essentially face the animator’s challenge when we try to find and use information on the Web. That’s because we’re often looking for things whose meaning, whose interpretation, demands focused human thought. We are looking not for business data, but for pieces of media, and the problem is that today, most of our searching has to be based on tags or brief textual descriptions that are associated with pieces of media, and not on the true meaning of the media itself.

The needs of the business world are not our needs.

It’s the subjective nature of media assets - this is what is at the heart of the problem facing us. Existing technology for searching the web is based on keywords and very short pieces of text.

There is other technology, though, under active development, stuff that serves as the information storage backbone of most commercial websites. It’s the technology that has for decades been used in-house (not on the Web) by businesses when they process large databases. But this stuff was designed to handle traditional business data forms, like integers, character strings, real numbers, dates, timestamps, and full text.

There is more, though. All of the major database management systems, along with tools for building and searching advanced websites are being retrofitted (or in some cases, built from the ground up) to manage more than keywords and text, more than standard business data.

But up to now, the focus has not been on supporting the kinds of information you and I are most interested in. The focus has been on extending database and Web technology to support xml documents, as well as more complex data objects, like those inside a Java program, as well as other forms of data found inside programs. This includes arrays and lists and short pieces of textual data, like the names of diseases.

In other words, we’ve been busy extending our support of the business world, so they can store complex business data in databases and make that information processable over the Web. You and I have largely been left out.

Finally, we are attacking our needs.

But there now many ongoing efforts to extend database and Web technology to make it useful to us. The new focus is on supporting blob and continuous media like images, video, and audio. This is extremely hard to do.

Why? Because the strongest means by which we deduce the meeting of business data is by looking at its internal structure and the terms that are used to describe that structure. A relational table named Prescriptions, with a character attributes Patient Name, Doctor’s Name, and Medication, and with a numeric attribute Dosage, is pretty easy to interpret.

But what do we do with a photograph, which is just a grid of pixels with no internal structure? Or a long series of images, along with a sound track, put together to form a piece of video?

The U.S. military has been pumping money into image processing for several decades, and so all is not lost. There is a vast body of mathematical research and software development that allows us to write programs that can find a particular face in a crowd and search satellite photos for airplane runways. But in general, we cannot at this time write a program that can process an arbitrary photo or video clip and tell us what it means. That means we can’t quickly search vast media database for useful pieces of information.

The goal behind the Semantic Web effort is to build a new generation of websites whose information can be searched automatically, and where information from multiple sites can be automatically integrated. To do this with numeric and character based data is quite doable. But when it comes to multimedia, like images and sound and video and 3D models and engineering designs, well, we have a long way to go. The meaning - in other words, the semantics - of these forms of data are complex and subtle, and highly dependent upon an individual’s interpretation of that media.

So, we see that we have only just begun our journey to create the new Web.

Feb 21 2009   5:19AM GMT

Namespaces and the Semantic Web

Posted by: Roger “Buzz” King
Web 2.0, Web 3.0, Web development frameworks, namespaces, XML

The Semantic Web. We introduced it in the entry before this one, which was the first entry in this blog.

This very aggressive goal, but if it were to ever exist, the Web would become something far more powerful than it is now. Today, we can only search the Web manually, interactively. We pull up Google, type in some keywords, and see what comes back. Then we begin to iterate. There is one obvious problem and two that might not be quite so obvious - and all three of them would be fixed if the Semantic Web really existed.

The key is that word “semantic”. It means that programs that search the Web, i.e., search engines of the future, would be able to search by the meaning or semantic content of the information we are looking for, and not simply by looking for keywords in the text of pages indexed by the search engine.

What are the three problems?

First, obviously, we would be able to perform a search with little or no iterating. This would radically reduce the need for a human to be in the loop, constantly guiding the search engine with more and more refined keyword searches. On the Semantic Web, a search engine would simply go out there and find whatever it is we need, and then deliver it up. If we search the Web by keywords and are looking for a treatment for tapeworms, we might not get the right results because we don’t know enough medical terminology to realize that we are looking for treatment for a disease caused by tapeworms, a disease called Taeniasis?

Second, not so obviously, the Semantic Web could come a lot closer to assuring us that our search was complete, in that the information returned was not only relevant, but that there wasn’t anything important that had been missed. If we are searching for treatments for tapeworms, and we find four possible treatments, how do we know there isn’t a fifth one out there that is more effective, quicker acting, and safer than the other four?

Third, and even more subtly, the Semantic Web would largely solve the huge problem of heterogeneity of data, of mixing information that isn’t truly comparable - essentially of mixing apples and oranges. Right now, when we search the Web interatively with Google, we might find one site that says that a “high end” notebook computer on one site would cost $3000, while we might find another site that says that a high end notebook computer costs $2300. Wouldn’t it be nice if the search engine could automatically ensure that we are comparing two computers that are truly similar in all significant ways?

So what’s behind the Semantic Web, what will power it if it ever emerges? A keystone technology will be that of “namespaces”. The idea is simple, and while it is only very much a partial solution, it is surprisingly powerful, given its simplicty. Essentially, a namespace is a collection of terms that multiple people agree to share, and furthermore, they agree on specific meanings for those terms. The Web, as it turns out, provides a powerful way of sharing namespaces: we can plant them on websites and anyone who wants to use those terms knows where to find them, along with their meanings.

One of the first namespaces to explode on the Web is called the Dubln Core. (Sorry, but the name refers not to the Dublin in Ireland, but to the Dublin in Ohio, where a group of people met to establish this namespace.) It is a collection of terms that can be used to describe resources that can be found on the Web, or in paper libraries, or in any other place where we store information. These terms include Contributor, Date, Publisher, Subject, and many more. And if you want to find the Dublin Core, it is publicly available at:

http://dublincore.org/.

We’ll look a lot more at namespaces in future entries in this blog. We’ll also consider such technologies as XML - the standard for specifying namespaces.

Feb 17 2009   11:22PM GMT

The difference between Web 2 and the Semantic Web

Posted by: Roger “Buzz” King
Web 3.0, Web 2.0, Rich Web Apps, Web development frameworks, Web development, Ajax, XML Schema, XQuery

The purpose of this blog is to discuss cutting edge technology that relates to Web 2.0 and the Semantic Web. What do these terms mean?

Let’s start with the definition of a third term. A Web Application is a website that provides some sort of substantive functionality other than simply filtering and presenting information. Evernote is a fantastic web app that stores your notes on a server, and allows you to create, group, and annotate your notes. Some folks say that a web app makes it clear that there is an application at the other end of your browser, and not just a bunch of static data. This is admittedly a pretty soft definition, but it’s reasonable. Another way to look at it is that a web app provides what would otherwise be a desktop application, but makes it accessible from a server so that users do not have to install and maintain an application.

So what’s Web 2.0? It refers to web development frameworks and tools that can be used to create highly responsive websites and web applications. AJAX does this, and the conical example people give is Google Maps. AJAX allows data to be retrieved asynchronously while a prior page is being displayed and manipulated by a user, and minimizes the amount of a web page that must be replaced with the next refresh.

A somewhat newer approach is embodied in Adobe Flex and Microsoft Silverlight technologies; in these cases, a web app is sped up by running more of the application’s logic inside a browser plugin (Adobe Flash or Microsoft Silverlight), rather than making the client machine (which runs the user’s browser) continuously talk to the web server. The overall challenge is to make web pages highly dynamic (meaning the data comes from a database and is not hard-coded in the web page) while giving the user response times that approach those of a desktop application running on a dedicated or near-dedicated machine. While this is intractable at this point, it’s a good thing to hold up as a goal.

The term Semantic Web does not narrowly refer to technology that speeds up response rates. Rather, it refers to a still emerging body of software tools whose overall goal is to automate the collection and integration of information gleaned from websites. The idea is to free the Google/Yahoo user from painfully interactive, highly repetitive keyword searches where we continue to hone our queries until we seem to be finding the right stuff.

Semantic Web technology includes namespaces, which try to put more smarts in websites by having data tagged with widely shared, standardized sets of tags. And things like XML Schema and XQuery can be employed to leverage namespace technology to support high-volume, set-oriented queries of data stored on web servers. These are very similar to the sorts of queries that can today be coded in SQL and run on single database servers running database management systems like Oracle, SQL Server, DB2, MySQL, and PostgreSQL. Essentially, XML-based technology takes the ability of a relational database schema to help us interpret data, and extends it to the entire web.

We will look at XQuery and XML Schema in future entries of this blog.

By the way, some folks are already talking about Web 3.0, which in many ways draws from both Web 2.0 and Semantic Web technology. We’ll look at this in a future blog, but a key focus is on making web apps highly multimedia.