mastodon.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
The original server operated by the Mastodon gGmbH non-profit

Administered by:

Server stats:

336K
active users

@bobwyman can you help me out? I'm trying to remember the term of art for the opposite of real-time search. Is it just "static search"? I feel like there's a better term for it.

@evan I've found that "real-time search" isn't really as useful a term as people would expect... But, if you insist on using it, then the opposite of real-time is going to be either "periodic," "repeated," or "queued," depending on what you're trying to say.

There are two kinds of search: Retrospective and Prospective.
1/ (see following notes)

@evan A retrospective search compares a query to one or more documents that have been previously indexed. Such a query tells you what has been indexed in the past. Because the query looks to the past, it is "retrospective." You can process that query in "real-time," like Google, etc. do, or you can queue the query for processing once or more times in the future, but the query will always be looking to the past. Most "real-time" searches are implemented as repeated retrospective searches.
2/

@evan A prospective search compares each member of a sequence of one or more documents to a previously indexed set of one or more queries. In such a system a query asks: "Tell me whenever!" Because each query looks to the future, the queries are "prospective." Many notification systems rely on prospective search. Similarly, much of the internal operation of SocialWeb systems can be modeled as prospective searches. (i.e. copy post to my feed "whenever" I am mentioned in it.)
3/

Bob Wyman

@evan The key to understanding the difference is to consider what is indexed. A retrospective system indexes "documents" which are usually just collections of static data. On the other hand, a query can contain ranges, boolean expressions, etc. which can't be indexed using the "inverted trees" and other methods typically used when indexing documents. Thus, true prospective systems, which index queries, require very different indexing structures than do retrospective systems.
4/

@evan Because most people don't understand how to index queries, and because this problem is typically not discussed in schools or in books on search, most systems that should use prospective search are actually implemented by doing repeated *retrospective* searches on incrementally built document indexes. While this is inefficient, it is easily hacked together and, given the power of machines today, most folk don't have enough queries to really notice the limitations of repeated retrospective.

@evan When the semantics of a prospective system are very simple, it is possible to use simple implementations. For instance, if the queries are limited to specifying a single string term (i.e. a user name) then one can build a simple, high-performing prospective system that extracts usernames from incoming posts and compares them to a hash-table of "queries" which are just usernames.

A proper system would allow Booleans, ranges, etc., but that requires a dramatic increase in complexity.

@evan Someday, I'll write this stuff up in a form which is useful for general consumption. It's really not that hard if you recognize that prospective search requires approaches different from retrospective. I've always found it odd that proper prospective search technology isn't more widely used outside large platforms like Google, Yahoo, Facebook, Bloomberg, etc. Many applications could much better serve their users' needs if they did this properly.

@bobwyman Until then, do you mind if I paraphrase for my ActivityPub book, with acknowledgment? "Prospective" and "retrospective" are way better terms than what I was using.

@evan I have no objection to anyone using these terms. Frankly I'd be pleased if more people knew the difference between retrospective and prospective search because almost all thinking about search today is about only half of the whole search problem -- the retrospective half. If more people recognized the distinction, we might get more people working on addressing that, as yet largely unexplored, half of the search problem.

@bobwyman @evan I think you have a typo in this message? You wrote "prospective" twice. Great stuff, I learned some new things!

@tedmielczarek @evan Thanks for catching that. I've edited the note to point out that people often use repeated *retrospective* searches to create the impression of having done a prospective search.