The eLibrary is a component of the Scoop technology SCS recently purchased. eLibrary is now being built on Linux. Scoop's eLibrary uses a proprietary search engine called Surfinity. This component was provided by Scoop only as a binary executable. It runs solely on Windows platforms, specifically the 13 year old Windows Server 2003. Surfinity is obsolete and no longer available for current operating systems. After the Scoop eLibrary source is converted to run on Linux, its search engine still needs to be replaced.
A current, supportable version of eLibrary is needed as part of providing a path forward for existing Scoop users on support. Further, it will be part of BENS (Best Ever Newspaper System.) It might also be leased as a stand alone product.
Spice's formula language (FL) supports database tables that have columns that are blobs (usually text fields.) Full text indexing and querying is built into the formula language.
The FL has been enhanced to support structured hierarchical objects (SHO). SHO are compound data objects that have a number of desirable properties. These extend beyond the relational database model currently well-supported by the FL. A SHO can be used to manage such multi-component things as advertising super orders and news packages as single variables or table entries.
SHO are implemented in the FL using JSON syntax. The FL compiler adds static type declarations to JSON. It enforces type checking for FL SHO and thus our use of JSON. This provides an important safety net for programmers.
JSON is the protocol used for passing data among web services. JSON also is the protocol used for SCS's evolving enterprise microservices bus, called MOAI. (Mother Of All Interfaces.)
(The MOAI is the key for moving beyond the linear workflows and data silos typically built into today's newspaper systems.)
We will deploy new search technology to support eLibrary and SHO in the FL.
SCS's current text indexing technology is built using CLucene, a somewhat old, C version of Apache's Lucene. Among search engines, it is regarded as small, efficient, stable and easily deployed. SCS usage includes full text indexing and searching of ads. It is also used as a fast secondary index within SCS/Track (CAS). It and its current usage will stay as is.
To support SHO and JSON variables in Spice FL databases, CLucene will be enhanced. So will Spice's FL. Setting up a Lucene index involves field naming and type specifications. The FL will use JSON to pass these mappings to a new CLucene front end. The same will be done with creating, reading, updating and deleting SHO index elements. The choice for the JSON syntax of these transactions will be explained below.
Apache Lucene is now provided in JAVA. It is often used as an embedded search engine in technologies such as Solr and Elasticsearch. Elasticsearch is currently the most popular search engine with over 50,000 downloads each month. (Over 20 million so far.)
Elasticsearch (ES) supersets Surfinity's functionality so it is a good choice for use in eLibrary. Putting ES (and thus eLibrary) on a dedicated search appliance isolates its huge footprint and technical complexity. This should make marketing and supporting SCS's search technology simpler.
The JSON syntax used within Spice's FL for SHO is to be a proper subset of the JSON syntax that ES uses. By this technique Spice programmers will be able to use the indexing tool that best scales to their needs. It also makes transitioning among search engines simpler.
Having eLibrary be compatible with ES will allow the many free open-source software (FOSS) query and interfacing tools that work with ES to be used with eLibrary and, perhaps, other SCS applications.
Elasticsearch is installed on SCS's networked systems. eLibrary now launches on Linux with access to Elasticsearch. The eLibrary conversion is well underway.