SMIL

There are Web apps and then there are Web apps.

In our continuing series on Web 2.0/3.0 and the Semantic Web, we have looked at one simple, yet impressive Web application, called Evernote. There are significant advantages of Web apps; in particular, the application is available wherever you can get onto the Web, you don’t have to run and maintain complex desktop software, and your data sits on a (hopefully) secure and backed-up data server.

Web Apps.

We noted that some Web apps, including Evernote, are both Web-based and desktop-based. Seemingly, this might be a disadvantage, because now, the user does have to install and maintain the desktop version of the app. But, in exchange, you have two copies of your data, at different physical locations. You also can use the app when you are not on the Internet. And, as far as Evernote goes, the desktop app is very far from difficult to manage.

Let’s look at this a little closer. Not all Web apps are the same. One problem is that too many vendors feel compelled to brag about the Web capabilities of their projects, and so we have to be suspicious - especially when it comes to older applications that have been retrofitted with Web capabilities.

Let’s look at a few applications. Please keep in mind that the first two applications are not advertised as “Web apps”. I am describing them only as a way of categorizing the Web capabilities of applications in general.

Minimal capabilities: exporting to the Web.

Our first example is an application that runs on Macs and is very impressive. It’s called Curio, and is made by a company called Zengobi. It gives you a workspace to which you can append text notes, lists, images, video, and sound clips. It also supports diagrammatic mind-maps. It’s great for a wide class of brainstorming techniques from simple note-taking to sophisticated workflow planning. It’s all-in-one nature makes it a little imposing and chaotic at first, but it is actually quick to master - and then its freeform nature proves itself to be very powerful. It is also very elegant.

Curio’s Web capabilities are extremely limited, however. All you can do is output a Curio file as a fixed HTML page. It cannot be updated over the Web. For convenience, it can export a file directly to your “Mac” Web account, if you own one.

Modest, often tacked-on Web capabilities.

Another example application is Filemaker. (I am referring to their products called Filemaker Pro and Filemaker Pro Advanced, since they are what I have used in my classes as the University of Colorado.) I teach database management systems, and I can say lots of good things about Filemaker. It is a very quick and simply way to get a full-fledged, scalable, visually-pleasing desktop database up and running. I like it.

But its Web capabilities are typical of applications that have added Web capabilities long after the fact. What you can do with Filemaker is “publish” a database on the Web, and allow Web-based updating and searching. It in effect turns the machine hosting the database into a simple server. But most of Filemaker’s capabilities are not available via the Web interface. And, the database only exists on its original site. All data remains there.

Native, full Web capabilities.

So, what’s a true Web app? I’d say it is an application whose native interface is Web-based, and where all or virtually all of its capabilities are available via a browser. Evernote is a good example.

There is a fuzzy line between “websites” and “Web applications”, as we have previously discussed. And in fact, some people consider virtually all powerful websites to be Web apps. This includes Amazon, Blogger, and Wikipedia, as well as countless lesser-known websites.

And, with respect to the deliberately narrow criteria we’re using here, these applications are indeed Web apps.

So, what characteristics do we see in applications that are powerful, and have native, complete Web interfaces? They are likely to store data persistently in a serverized database management system like MySQL, and present the user with web forms to fill in, and return to the user dynamic Web pages populated from the database. A website that we might be willing to label “Web 2.0″ would be one that is highly responsive and manages large amounts of data.

We might call it Web 3.0 if it also manages large volumes of continuous data (like audio and video), and presents to the user a highly multimedia web interface. But these terms are vague, and drawing lines between them is to a certain degree misleading and a distraction.

Perhaps something that might be a truly Web 3.0 characteristic is that the application, rather than just delivering up video and audio, uses a combination of multiple forms of media, in concert, to interact with the user. We looked at SMIL, an XML language that allows the user to build presentations that coordinate multiple forms of media, such as images, sound, and video. The SMIL programmer can arrange media on the screen, and specify how the various pieces of media will be displayed over time.

Glide: the Web-based desktop.

But let’s look at one very, very aggressive attempt at a true Web 3.0 application. It’s called Glide, and you can get yourself a free account. This application does not support any sort of desktop-based version, and so you do have to be online to use it. It also needs a very fast Internet connection, because of the wide variety and high volume of data it allows you to manipulate.

What’s Glide? It is advertised as “the complete mobile desktop solution”, and it provides a complete, virtual, web-based computer. With it, you can edit photos, draw diagrams, store media files, send and receive email, manage a calendar, manage video, write documents, even build a website - in other words, do almost everything a non-programmer might want to do with a computer.

Its interface consists of three main windows. One is a virtual desktop, with various applications ready to use; another is a portal where the user can access the Web and develop websites; the third is a virtual hard drive, where media and files created by the various applications can be stored and accessed.

Is this the way of the future? It completely frees a user from having to buy, install, and maintain complex, expensive applications, although you still need a computer with a browser to run it. One drawback is that none of its apps, as near as I could tell, can compete with the dominant desktop applications. It is not Photoshop, it is not Dreamweaver, and it is not MS Office Outlook. But its apps are not trivial: they do the job just fine. And the entire interface is simple and visually pleasing.

There is also a way to sync your files on your desktop with the files on the Glide servers, and their documents and spreadsheets are apparently compatible (to some degree) with Microsoft’s Word and Excel. But they apparently are not planning on creating any sort of hybrid web/desktop based product. Glide’s goal is to move us all toward the Web and away from our desktops.

The Glide servers seemed fast enough to me, by the way. That’s the big question. Can it be as responsive as a desktop computer? Well, it’s as fast as my Vista machine… But slower than my iMac.

Give it a try.

Full Text searching: cleaver heuristics for managing large web-based document collections.

There is an explosion of technology for supporting sophisticated forms of media on websites and in web applications. In our continuing series on advanced web applications (in particular, as they pertains to the Semantic Web and Web 2.0/3.0), we’ve looked at continuous media, in particular, video and multimedia presentations. But there is a very old form of continuous media, something that is perhaps the dominant media on the Web, and that’s text.

It’s becoming a very major issue in web development.

Text.

In this blog entry, we’ll be looking at a particular form of text, called “full text”.

But just what is text to begin with? It’s character-based data, anything we can read.

And what will we want to do with it in next-generation web applications? It’s important to note that more and more vast libraries of documents are being put online. Web applications need to provide far faster and more accurate searches of documents than what we can perform with Google.

Interestingly, a successful technology, called “full text retrieval”, is already in place in the relational database systems that underlie modern web applications. It’s there working for us, and we are likely to not be aware of how clever it is.

It’s also something that should be used much more heavily by web application developers.

Let’s step back and consider three different - and increasingly more sophisticated - ways of managing character data.

Atomic Character Attributes.

First, there is the traditional relational database approach, whereby data is stored as tables made of rows of atomic, fixed sized attributes. By atomic, we mean that each attribute has no internal structure. So, a table of insurance claims might have rows with the following attributes: Claims-ID (an integer), Amount (an integer), Medical_Problem (a fixed length character string), and Subscriber_Name (a fixed length character string). Using SQL, the universal database “query” language, we might look for all rows that contain the name “Fred Jones”. Or, we might search for all rows that have claim numbers that are between 110 and 115.

Essentially, this approach limits us to comparing small strings of data to each other or to fixed values. There are some common extensions that we find in relational databases, such as being able to ask the question to find all rows where the Medical_Problem is something like “broken leg”. Then if a row actually has the value “broken legs”, we would most likely see this row in our results.

Full Text.

Second, there is the ability to search pieces of text according to their natural language (in this case, English) meaning. In this case, we consider the character data to have internal structure, and the values are not considered atomic. Often, these pieces of text are long and of variable length from one row to the next.

It is actually an extension of - but a very dramatic one - of the like operator in SQL.

It is what we call “full text” management or retrieval, and modern relational database management systems like MySQL and Microsoft SQL Server support this. This was seen long ago as a critical extension to relational database technology. Thus, we might rename our Medical_Problem field to Doctor’s_Diagnosis, and allow free form English text in this attribute, as well as allowing the value to be quite long. Then we might search for all rows where the doctor describes “fractures of the lower limbs”. Notice that none of these words might actually appear in the attribute, which might simply refer to “broken legs”.

Natural Language Processing.

This capability would clearly be very powerful, if we could do it right. The problem is that to support it fully, we would need to use highly advanced natural language processing techniques, which are very time consuming to execute, especially on huge databases of large documents. The full text approach tries to simulate true natural language searching in a far less expensive way. The real thing, by the way, might not be all that accurate anyway. Natural language is naturally ambiguous and very subtle.

True natural language searching would be our third way of processing character-based data, by the way. It is not a fully developed technology. And importantly, we usually don’t need anything that fancy.

The Clever Compromise.

So, our middle option, full text searching, is what dominates today - and it is a surprisingly accurate, and efficient, technique that operates on a small set of heuristics. It can transform a dumb webpage where we can only search for small, fixed character strings, to a rich next-generation webpage that can effectively be searched according to its meaning. It allows us to manage very large text documents in web applications - and get us surprisingly close to the semantic power of true natural language searching.

We’re not going to go into a lot of detail here, but here are some of the heuristics that are used in full text search. First, “stemming” and related techniques are used; they conjugate verbs, detect plurals of nouns, and remove prefixes and suffixes. Another technique is to use a “stop list” that lists words that should be ignored, like “the”. The system might also let us specify the “proximity” of words; this refers to how closely specific words should appear in a document. It can also be powerful to include a synonym checker. And the ability to allow for “wild cards”, in particular, letters that may vary in a passage without changing its meaning, can be quite useful. Dictionaries of technical words that pertain to specific domains (like medicine or law) are very useful. We might also provide a feedback capability, whereby users can train full text search engines to be more accurate.

This clearly doesn’t come anywhere near true natural language processing - but it is fast. It will be a growing technology on the new web, with a lot of hidden development, making this heuristic-based technique more and more effective.

Indexing.

We should note that there is a significant up front cost in preparing a document for full text searching: we need to build an index with an entry for every (non-stop) word in the text. Then, when a query is executed, we can look for words in the document by searching the index, instead of searching the full text. If there were no index, the search would be extremely time-consuming.

The Future.

As more and more governmental, educational, medical, and other complex documents become available on the web, advanced full text searching will enable us to search vast databases in a tractable fashion. Even more clever full text retrieval engines will turn dumb, “gotta Google them” document portals into true Web 3.0 and Semantic Web applications.

SQL and XML: declarative is exciting

In the continuing series of blogs on the Semantic Web and other advanced web technology, we’ve looked at XML as a cornerstone of the technology that allows us to markup data, and in combination with namespaces, create powerful tools for sharing the meaning - and not just the structure - of data. There’s something special about XML that is at the core of its truly amazing widespread adoption, that explains its versatility as the language of choice for tagging data, no matter what the purpose.

XML and its powerful children

A key purpose of this blog is to provide a continuing examination of the Semantic Web - and certainly one of the most critical technologies to discuss is XML. Why is it so important?

First of all, just what is XML?

XML stands for eXtensible Markup Language, the extensible part is the key to its power.

Markup Languages.

Let’s step back though and look at the markup part first. “Markup” refers to the process of embedding commands in data. HTML is a markup language. When a browser fetches a web page from a web server, it processes the text-based HTML “markups” that appear in the page in order to present the page to us.

HTML: a Markup Language.

Importantly, HTML is focused on the visual appearance of information. It controls the layout of web pages, including “controls” such as menus and buttons. It also allows us to link pages together. One of the biggest jobs of HTML is to tell the browser how to layout pieces of text, such as the descriptions of books sold by Amazon.

HMTL has a fixed set of legal tags. Here is a sample HTML file:

<html>-
<body>
<h1> This is a heading </hl>
</body>
</html>

Notice that every tag comes in pairs, one with with a “<>” and the other with </>.

This HTML opens by telling us that it is an html file. Then it says there is a body to the file, and that there is a heading to be printed. This file will print the words “This is a heading”.

The important point, though is that these tags - html, body, and h1 - are HTML specific tags, and we cannot invent our own.

XML: a Far More Powerful Markup Language.

Now, let’s see what happens when we can invent our own tags.

XML is also a markup language. It was developed as a way to embed markups in data, so that the meaning of information can be communicated. In order to do this, XML allows us to do something we cannot do with HTML: we can specify our own “tags” so that we can add a lot more expressive power to our markups. There are two particularly critical aspects of XML tages.

The first is that there are two main sorts of tags: “elements” and “attributes”. Elements can have complex structure, and in fact, we can embedd elements inside elements. Attributes are simple values and have no internal structure.

The second critical thing is that we can use words taken from shared namespaces as values in tags in XML. This gives XML the power of shared, detailed terminologies that are available globally via the web.

The real power of XML is this: we can produce our own extensions of XML by defining our own tags. Each of these extensions is itself a complete markup language.

This is why it is such a critical part of Semantic Web technology: we can use it to capture the meaning (or “semantics“) of data so that it can be processed automatically. HTML controls the way a page is displayed only, and we have to use our eyes and minds to interactively interpret this information. But XML can be interpreted by a program, thus allowing powerful, automatic searching of the web.

An Example of an XML Extension.

Let’s look at an XML-based language, in particular, at its use of elements, attributes, and values from a shared namespace.

Below is a piece of code written in an XML extended language called SMIL. SMIL allows us to create multimedia presentations, with various pieces of media laid out on the display, as well as being sequenced in time.  (SMIL stands for Synchronized Multimedia Integration Language.)

First, let’s start with the core of a SMIL program:

<smil>
<head>
<layout>

… here is where we put commands that control the visual layout of the page we are constructing with SMIL …

</layout>
</head>
<body>

… this is where we put the core of our SMIL program, the part that specifies the multimedia presentation that is to appear in the page …

</body>

</smil>

Here is the entire program, fleshed out:

<smil xmlns:qt=”http://www.apple.com/quicktime
/resources/smilextensions” qt:autoplay=”true” qt:time-slider=”true”>
<head>
<meta name=”title” content=”Buzz’s Video”/>
<layout>
<root-layout background-color=”white” width=”320″ height=”290″/>
<region id=”videoregion” top=”0″ left=”0″ width=”320″ height=”290″/>
</layout>
</head>
<body>
<seq>
<video src=”http://files.me.com/kingbuzz/radljq.mov” region=”videoregion”/>
<video src=”http://files.me.com/kingbuzz/radljq.mov” region=”videoregion”/>
</seq>
</body>
</smil>

We don’t need to worry about the specifics of this code. The values of attributes are in quotes, and the values of elements are inside <> and </>. So, background-color is an attribute, and video is a element.

Let’s look at the beginning of this program:

<smil xmlns:qt=”http://www.apple.com/quicktime

/resources/smilextensions” qt:autoplay=”true” qt:time-slider=”true”>

This code refers to the SMIL extension, i.e., namespace.  That’s what xmlns stands for, by the way: XML namespace - i.e., the set of attribute and element tags invented specifically for the SMIL extension to XML. By pointing to this namespace, our program identifies itself as being a legal SMIL file, and this tells Quicktime, which can play SMIL files, how to interpret it.

To see this, do this: Download Quicktime from Apple, if you don’t have it. Then put the above program in a file called buzz.smil. Then open buzz.smil with Quicktime.

Quicktime will read the file, locate the SMIL namespace on the web, then read the tags inside the SMIL program, and use them to interpret the rest of the code. This will direct it to download a video from my site - an excellent piece of animation built by one of my Intro to 3D Animation students, named Jochen Wendel. And in fact, Quicktime will play it twice - that’s that the <seq> </seq> tags mean: play it twice in sequence.

The Exciting Part.

Do you see what happened? We used a predefined namespace belonging to the SMIL extension of XML to write a program that can find a video, download it, and play it twice!

Why do we care?  It’s not that building a language to play various pieces of media, like Jochen’s animation, is a big deal in itself.  It’s that XML is extremely versatile.  By defining a set of tags and then sharing them, we can embed within information the means for interpreting it - and thereby create an endless array of powerful languages.

This is very important. XML and its powerful children (such as SMIL) are changing the web in a big way.

There’s something  else, too, something that is equally important.  XML is declarative. We’ll look at this in another blog soon, but essentially it means that an XML language like SMIL is easier to read than imperative code, like Java or C.  Look at my SMIL program, and then look at a Java program.  Which is easier to understand?  We’ll get to this.

Multimedia, what is it? Why do we care?

This is the fourth in a series of blogs on the Semantic Web and Web 2.0/3.0.

To get us going here, just what is “multimedia”? At one level, it simply refers to applications that manipulate, store, and/or present multiple kinds of media, such as text, video, relational data, sound, animation, etc. More pragmatically, it refers to the introduction of blob and continuous forms of data into applications that traditionally manage simple data, like character strings and numbers. In its most aggressive form, multimedia refers to the sophisticated integration of traditional, blob, and continuous data into integrated data forms that convey their own semantics.

A quick note: Blob data is data that is stored in a semantics-less fashion, usually as simple binary or character data. This could be almost any sort of data, such as images, video, sound, or natural language - but the key element is that the language or system being used to manipulate it doesn’t have an appropriate, specific data type. Blob data is often large, of variable size, and usually requires a sophisticated, outside application to interpret and present it. It is the default, catch-all way to store advanced forms of data in relational database management systems.

Another quick note: Continuous data is data that has a temporal aspect or can be broken down into segments that have their own identity. The visual part of video can be broken into clips; in fact, it can be broken all the way down to individual pixel-based images. Sound can be cut into pieces. James Joyce’s Ulysses is a big piece of continuous textual data. Like blob data, we typically need an outside appliation to interpret it. (Even the most complex application, a human, generally has trouble doing this with Ulysses.)

Back to the Semantic Web, Web 2.0/3.0, and multimedia.

In a previous blog, we tried to define these two terms and explain why they are very different concepts. The Semantic Web is an attempt to automate the searching of the web and the integration of data collected on the web; the idea is to greatly ease the painful interactive nature of using a search engine like Google. Web 2.0/3.0 (and no, there is no sharp distinction between the two) are largely about performance, of making web applications as responsive as possible, potentially as responsive as desktop applications.

But they share one common goal: effectively managing advanced forms of media. From the Semantic Web perspective, how can we search things like sound, images, video, and natural language in a semantically-meaningful way? We use sophisticated tags and image/sound processing to do this, but it is only a small step toward a solution.

From a Web 2.0/3.0 perspective, how can we deliver up such forms of media in way that is highly responsive? Video streaming on the web is a huge challenge, for example. Or, how can we interact with video in a responsive way, in such things as games and digital libraries?

The web, in fact, is inately multimedia: we take images, icons, links, text, video, sound, and various user controls like buttons and menus, and put them together in highly sophisticated ways. And behind these web pages, databases often sit, populating dynamic pages with information in response to user requests - this data is virtually invisible to search engines. This is what makes the Semantic Web in particular such an incredible challenge. How can we ever hope to search the web automatically?

There are modest advancements that have been made. One example is something called the Synchronized Multimedia Integration Language (SMIL, pronounced like the facial phenomena). It is an XML extension that supports basic constructs to glue multiple forms of media together in two dimensions and in temporal sequences. Using XML elements and attributes (the basic constructs of XML), we can create multimedia presentations in a precise, unambiguous way.

SMIL presentations can be processed automatically. This is very significant.

So, multimedia: it’s at the core of both the Semantic Web and Web 2.0/3.0. It is one of the basic motivations for their existence.