Searching by semantics.

Here, we look closely at one specific issue related to managing complex media: How to categorize and search advanced forms of media by their “meaning” or “semantics”. This is extraordinarily difficult, and in fact, in general, it is impossible. This is why we usually rely on relatively low-level heuristics and can only simulate search-by-semantics in simplistic ways.

Consider a library of soundless video clips. Let’s assume there are many thousands of them, and they vary in length from seconds to hours. First of all, the only clips we can afford to download and actually view in real time are the ones that are only seconds or minutes in length, and we can do this only if we are somehow able to limit the search space to a small handful of candidates. Keep in mind that a video can consist of twenty to forty images per second.

So what do we do?

Searching previews.

We could search tiny samples of our video clips, perhaps taken from the beginning, the middle, and the end of each clip, but this doesn’t actually well, either. We need something that can scale, that is automated.

Searching tags.

The dominant technique is to extract information concerning low level attributes of the video clips (such as their format and pixel count) automatically, and then have experts add more tagging information by using widely adopted, formal namespaces. We might use a geography namespace to mark clips as having rivers and mountains in them.

These two forms of tagging information might be encoded together using the very popular MPEG-7 language. This creates a very indirect way of searching video clips. We don’t actually search them. We search the hierarchically constructed MPEG-7 tag sets that describe the videos. This at least allows us to use SQL in a reasonably straightforward way to do the searching.

Searching for specific images.

There is very good technology for processing images for fixed pixel-based subcomponents like individual faces. We can also search for video clips that have any faces at all in them.

In general, it’s easier to search for things made by people because they tend to be more angular and regular in shape. These include specific buildings and types of aircraft.

Searching for colors and shapes.

We can also search for more abstract subcomponents of images, like polygons, circles, and the like. Despite the fact that video images are pixel-based (or “raster”), there is good technology for isolating the lines that form the boundaries of subcomponents.

And we can look for colors and compare the relative location and dominance of various colors, like images where 63% of them are a particular shade of orange.

Searching for change over time.

We can also search for pattern changes in the series of images that make up a video clip.

But none of this has much to do with the real meaning or semantics of images and the video clips they form. Taking this next step is huge challenge.

Semantics.

How can look for a setting sun or a ball moving across a tennis court, without knowing the details of the sunset or the particular tennis court in advance?

We can use the colors and shapes approach to look for a big orange ball descending below a possibly-jagged horizontal line. We could look for a small, white or yellow spherical object move across a big green rectangle.

One way to raise the bar a bit is to use domain-specific knowledge about the images being processed. It’s a whole lot easier to spot that tennis court if we know that’s what we’re looking for. Then we can fill our searching software with lots of detailed information about the various sorts of tennis courts. We can also more easily isolate the tennis court in a larger image if we know it’s there somewhere. This gives us an extra edge, so we can perhaps find the court, even if it turns out to be brown and not green, or if the surrounding terrain is almost the same color as the court.

We of course never get away from searching by heuristics that only simulate the process of determining the true meaning of a series of images. We can never truly search by semantics.

But we can do something else: we can get humans into the loop and train our software to do a better job. We’ll look at this next.

Relational database management systems, such as MySQL, Oracle, MS SQL Server, DB2, and Postgresql, support the relational model. A database is broken up into tables, and each table consists of rows. Each row is a series of values. A row in a table called Insured Drivers in a motor vehicle database might consist of:

Fred, 2010 Toyota Prius, State Farm Insurance, 1112233444.

1112233444 might be a unique identifier that the government assigns to each driver. This would be the “primary key” for the table Insured Drivers. The point is that human names are not at all unique, and so in relational databases, we introduce artificial keys in order to disambiguate queries. We still need the value Fred in the row because we want to know how to address him with a letter or email.

Problems with relational databases.

There are a few critical points to note with this approach. First, such a simple way of representing data allows the database to quickly deliver large sets of rows from this table to the memory of a computer, so that they can be effectively searched in bulk. We might want to know the names of all people who drive a Toyota Prius and are insured by State Farm, for example.

Another thing is that we might like to be able to put more complex items in a row. We might want to have another value in a row, one that gives a driver’s address. But an address has a few parts to it, and is not itself a simple value like a name or a car model or the name of an insurance company.

It is important to also note, however, that relational databases do indeed support the creation of more complex values, such as an address. But the more complex values we put in rows in tables, the harder it is to read in a large number of rows at once.

In fact, we could create a value that represents a very complex object, one that refers to rows in other tables. For example, we might want to replace the value Fred with a reference to a row in another table called Licensed Drivers, because there is a lot we might want to know about Fred, other than just his name. But then it would become very difficult to read in lots of rows of a single table quickly.

It might be that if we follow a link to another table that describes drivers, these rows might themselves have links in them, thus allowing a value in a row to actually consist of an object, like we would in Java or C++. And in general, these links between tables could be chained together, and extend arbitrarily far. Do we chase all of these linked references down for every row of Insured Drivers, or do we not follow any of these links so we can read in a large number of rows? Then we would worry later about getting more information on each driver.

Importantly, relational databases are still very much the dominant database technology in use in businesses and other organizations, as well as on the Web. We need to keep in mind that we have already aggressively extended them by supporting values that have internal structure (like addresses) and with the ability to create complex objects (like drivers). How far do we go in extending them?

Where we stand today.

Indeed, the extensions we have already made to relational databases have created a serious optimization problem.

But it’s worse than that. Here’s something else to consider. Relational databases were born into a world where flat business data was pretty much the only game in town. However, relational databases are being asked to manage far more sophisticated forms of data, like photos and video clips and voice tracks. There are a couple of problems that crop up. First, a row with a video clip as a field could be huge. We might only be able to read in a single row at a time and this could make searching an entire table intractable. Worse, how do we even search for rows that contain certain pieces of video? How can we search for all video clips that show Fred getting into a car accident?

Where to go from here.

In previous postings of this blog we have looked at media databases, and in particular, at techniques that can be used to tag complex forms of blob and continuous media (like photos and video clips). What’s important to note, though, is that there is a major dilemma right now in the world of database software. Can we continue to shoehorn more and more complex forms of data into relational databases, or do we need to throw in the towel and start over?

More on this next time…

Managing advanced forms of media, such as images, sound, video, natural language text, and animated models have been discussed a number of times in this blog in the past.  Traditional information systems, such as relational databases, have been engineered largely to handle the sorts of data we have in business applications, primarily simple numeric and character string data.  To the SQL database programmer, the nice part is that the data speaks for itself.  If a field is called Name, and the value is Buzz King, the semantics of “Buzz King” is pretty obvious, and it can be processed in a largely automatic fashion.  The same goes for a field called Age, with a value of “97″.  

Searching advanced media: far, far more difficult.

But modern media is far more complex than this.  ”Blob” data like images, and continuous data, like sound, video, and natural language text, are very difficult to search and interpret automatically.  There are two approaches that have been taken to resolve this dilemma.  

Tagging: the simple approach.

The first is tagging.  Descriptive terms, often taken from large, shared vocabularies, at attached to pieces of media.  These vocabularies can be very domain-specific, dedicated to areas like medicine, law, and engineering.  

Intelligent processing software: the second approach.

The second technique is the automatic processing of pieces of media using image processing, natural language, and other highly intelligent software.  These applications are very sophisticated and understood only by experts.  And, these applications often demand a lot of processing time, and this makes bulk processing impossible. It’s also true that the results can be haphazard.  Some pieces of media can be interpreted precisely, others not so precisely – and dramatic mistakes are frequent.  A tennis court might be mistaken for an airplane runway.  There’s a huge trust factor involved in cranking up image or sound processing software or natural language software.  

Often, we can provide feedback so that these applications can learn, over time, the way we want media to be interpreted.  We can help the software learn the difference between a tennis player and a member of a ground crew on a small runway. All of this is hugely expensive, in terms of the cost of developing the software, and in terms of the physical resources needed to run the software.

A middle ground?  Not really.

So, is there some middle ground?  Something simple, yet more “intelligent”?  Yes, and the answer is to take a sophisticated approach to what otherwise might be very simple tagging techniques.  However, the core problem with tagging remains: we search and process tags – and not the actual data.  It is an indirect, but fast process.  The goal is to come as close as we can to simulating the results of such things as image processing, but to do it with a simple, yet comprehensive, accurate tag-based technology.

We’ve looked at some of the solutions that have been proposed.  They include Dublin Core, MODS, and MPEG-7.  The first is very simplistic.  The second is more sophisticated, in that the terminology used is broader and far more precise.  The third is very aggressive in that it supports the complex structuring of tag data elements.  

So, what are we really doing?

In essence, we build a hierarchy of metadata and then instantiate it for every piece of media we want to catalogue and later search.  What we are doing is creating a parallel database, one where every piece of blob or continuous data is accompanied by a possibly very large tree of structured tagging information.  The parallel database has its own schema and an instance of it is created for every piece of media in the original media database.

The end result?  Instead of creating some sort of media-centric query language, like an SQL-for-video, we give up on trying to search the media database itself.  The query language remains largely ignorant of the nature of blob and continuous media.  We can continue to refine and expand the schema of the parallel database until search results are satisfactory.

More later…

From the last posting.

Here is a piece of the RDF code from the previous posting:

<rdf:RDF

xmls:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”>
xmls:zx=”http://www.someurl.org/zx/”>

<rdf:Description

rdf:about=”http://www.awebsite.org/index.html”>

<zx:created-by>http://www.anotherurl.org/buzz</zx:created-by>

</rdf:Description>

</rdf:RDF>

This can be interpreted as the webpage at awesite.org/index.html was created by Buzz.

Another representation of RDF-based information: 3 triples.

We see from the above that RDF simply represents triples. We could simplify it even more as:

http://awesite.org/index.html was created by Buzz

Part of the reason that the original RDF code above is so much more complex is that the full syntax lets us specify that we are using terms that are defined at specific web addresses. This allows people to use standardized terms and greatly enhances the specitifity of an RDF specification. The full syntax also allows us to reference pieces of information that reside on the Web. (See the previous three postings, 1, 2, 3.)

Before we launch into a SPARQL example, we need to make an important distinction between syntax and symantics. The code above is written in a particular syntax for RDF, one that uses XML. We note that because syntax needs to be very precise, it tends to be verbose. This can cause syntax to obsure the conceptual simplicity of underlaying semantics, or meaning.

But this isn’t the only way to specify RDF triples. Let’s look at some information that is much simpler, and at the same time, let’s look at using a different syntax for specifying RDF-like triples. Here are three triples:

<http://awebsite.org/ > was-created-by “Buzz”

<http://awebsite.org/ > was-created-by “Suzy”

<http://anotherwebsite.org/> was-created-by “Alice”

This is a very simple program. It consists of a two triples that say that a website named awebsite was created by Buzz and Suzy, and another triple that says that Alice created a website called anotherwebsite. We are not saying that was-created-by is a widely used term; it may have been invented only for particular RDF specification, and its meaning would therefore not be precise. We can only interpret it from our general understanding of English words. We also have no idea who these people Buzz and Suzy and Alice are, and we have no other information about them.

SPARQL: searching triples distributed across the Web.

Now, here is a piece of code:

prefix website1: <http://awebsite.org/ >
SELECT ?x
WHERE
{ website1:was-created-by ?x }

We’re getting very close to real SPARQL, by the way, and if you know SQL, you can see the extremely similarity. But syntax is not our issue here. We’re trying to look at concepts.

This code will find the creators of http://awebsite.org. You could imagine that there are actually many thousands of these triples, and that they tell us who built a large number of different websites. Now, we see the power of this query. It will search through all of these triples and find the two of interest to us, and then pluck off the names of the creators.

In fact, these triples could be distributed all around the Web, and we could imagine a search engine taking this query and running it everywhere on the Web where was-created-by triples are stored, and then having it bring back all the creators of awebsite, even if there are a hundred developers, and even if these names are spread around the Internet.

Next, the bigger issue.

In the next posting, we’ll look more closely at SPARQL. One thing we will consider is why it does look so much like SQL. There is a powerful reason for this that has to do with searching information in general.