The death and REBIRTH of the Scoop™ Newsroom System
The current situation:
Martha and I have some news for you about the future of the Scoop newsroom system SCS supplied you and that we have supported and you have used for so many years.
We call what's installed at various SCS Scoop sites Scoop 5.5. (Or Scoop 5 for short.) And the short, sad story is that Scoop 5 is reaching end of life. While all the good things of Scoop remain good, the bad stuff is getting out of hand. Moving to workstations running Windows 7 and later saw cracks in the GUI. (E.g., drag ndrop seemed to have dropped off the face of the earth.) Mac and PC versions of the clients could never seem to stay in sync. Especially troublesome was the lack of plugins that were compatible with current versions of InDesign and modern workstation platforms. What we've heard from many sites was "We are just sitting on it."
Believe me, we tried to fix this. We shelled out big bucks to buy Scoop Publishware's Scoop 5.5 source code. What a fiasco! I agreed to take the code "as is". Only the real situation was the code was "as isn't".
We tried to build it from source and found that many of the "module libraries" (i.e. the components that WinEdit, etc. used) were a mix of custom code, Borland code, and licensed, proprietary third party binary components. These weren't compatible with any build tools, either current ones or legacy ones.
One typical and disturbing situation had to do with eLibrary. We all know how valuable your library is. At some of our sites it is the only recorded history of the local community. You could try to replace the application, but what do you do about the database? So way back Scoop licensed Surfinity for the library's full text indexing engine. A nice tool for 1999, however what about today? Where is its source code? We never found it. We couldn't even find the Surfinity company. We were really stuck in a software swamp.
BTW I told our daughter, Sharon, about this and she said, "Dad, you at SCS are so lucky. You keep complete control of and distance from the platforms you use. Most of the rest of the software world makes junk like you are seeing. If it runs, just ship it and let others worry about ongoing, long-term support."
The issues with the Windows GUI were severe. We talked to the former Scoop team members. They said that Scoop's drag and drop problems lie deep in the internals of the proprietary Microsoft's foundation classes that Scoop 5 uses. It is impractical to try and fix them.
Ulf Wilkenson, formerly Scoop Publishware's owner, once employed a team of experienced and competent Swedish developers. Things began falling apart when he fired them and contracted with an overseas development company to save money. They were responsible for the failed Scoop 6 effort.
This isn't the first time I've seen offshoring development kill a product or even a company.
There's a phrase that begins "Up the creek..," that
comes to mind. The situation with Scoop 5 is intolerable.
Reject the old ways.
When we think of Scoop 5, we see something that clashes with our culture and business model. If you really checked it out, you would agree with us.
It allowed five storage management solutions: the file system (i.e., no database), MS SQL, a licensed, relational database management system (RDMS) from Microsoft and only for Microsoft OS’s, Oracle SQL, also proprietary and, alternatively, both My SQL and PostgreSQL, popular, platformi ndependent, free and open source RDMS’s.
Why you would want to offer five mechanisms when one will do is beyond me. Especially when two of them require costly licenses and restrictions on resellers (like us).
The information retrieval storage system from Surfinity, which was used by eLibrary, was the source of its own special problems.
Then there was the offering of both InDesign and QuarkXPress support. Well, you know what happened to QuarkXPress. I don't know a single reseller or XTension developer that liked dealing with them. We certainly didn't.
You can't imagine how both Mac and Windows client workstations were supported. There were two separate programs and two separate development teams. The Mac guys were quite good. The Windows team, not so much.
Not only that, each required special coding to use the proprietary GUI tool kits each had.
I could go on for hours. Scoop 5 can't be where we start from if we want to go anywhere.
The proposed future:
We took a bold step. We forswore any attempt to incrementally progress step by step from the existing (almost) working Scoop 5 system and its code base. To do so would just have placed us in the position of being upwardly compatible with previous errors. And forever fixing ancient bugs.
The first step was to say goodby to QuarkXPress, then the file system and MS SQL or Oracle for data management. Junking Surfinity was a nobrainer and so were platform dependent GUIs and proprietary operating systems.
What a relief! Now we could concentrate on doing the good stuff. The choice of platform was easy, just use Linux on NUCs, like what our current low cost, high performance advertising systems run on every day. Or support cloudbased deployment as an alternative. Scoop 7 uses TLS (Transport Layer Security) to encrypt all communications between the server and the client. TLS is the same technology used in web browsers to provide the "S" in HTTPS. Effectively this means that, if properly configured, any and all data exchanged through Scoop cannot be read by any eavesdroppers.
Kicking out Surfinity was easy. We substituted ElasticSearch, currently a very popular FOSS search engine. Unplug Surfinity, plug in ElasticSearch and voila, searches are better, more intelligent and faster. And it's free. No wonder Surfinity is gone. BTW We have built ElasticSearch databases with over a terabyte (over 10 year's worth) of data from Scoop newsroom systems.
One of the ugliest parts of Scoop was its rights management technology. Not only could we not find all of it, what we did find sucked big time. We junked it all. If you want to keep someone from using software, it should not be done module by module, but at a higher level. We've never done this before and don't want to start now. What if it breaks? What kind of vendor would be willing to jeopardize a customer's publishing for a payment?
To unify the GUI across workstations, be they Macs, Windows PCs or Linux, we chose Qt. Qt is a GUI toolkit that allows a single code base to be used for multiple platforms. Qt is dual licensed under both open source and proprietary/commercial licenses. We are using it under the free, open source (FOSS) license.
With our unification of all database access through an SCS developed front end to PostgreSQL, we could hide the differences in file and operating systems. Qt helps with this and many other multiplatform issues.
Just how special is Qt? Well, it has builtin components for internationalization that allow a product line to be deployed using multiple languages and localizations. Everything from alternate strings (words and phrases) to how dates, times, numbers and money are displayed can be done through Qt. What a super tool!
Here's the surprise!
We have a partnership to announce.
There was new coding done after Ulf sent us the source we bought. It was done by his former 25% partner Juha Siintola (Jussi) in Finland. Jussi bought out Ulf, acquiring all of Scoop Publishware. On Thursday August 3, 2017 we inked a deal with Jussi that changes everything.
Our agreement gives us joint copyright ownership of the Scoop 5.5, 5.7 and 6 code (and now Scoop 7) and the rest of the intellectual property of Scoop Publishware.
Jussi will supply development resources focused on InDesign plugins. We will focus on the rest. We will not compete with each other. Our territory is the Western Hemisphere and his is the rest of the world.
We will be setting up Scoop to run for those who speak Spanish, French and Portuguese. SCS already has customers in 21 countries, including ones in North, Central and South America and the Caribbean, so the internationalization is appropriate. (William knows all these languages, plus about a halfdozen more.)
With the partnership with Jussi, we start with installed bases of Scoop in over six countries. Clearly the system is well liked. It's just suffering from bit rot.
Going forward, Scoop 7 is in alpha release. Interestingly, it currently has about 75% of the functionality of Scoop 5 and yet has a code base of about 50,000 lines of code. Scoop 5 has over 450,000 LOC. What the heck!
Call Kurt Jackson (610 7467700) to talk further. He can arrange demos and provide additional information.
Platform independent applications
We work very hard at SCS at maintaining platform-independence in our applications, and we have a long history of this. Years ago, we delivered systems on DEC equipment running RSX, RSTS and VMS; HP equipment running MPE; and IBM equipment running MVS. More recently, we have delivered systems running under SCO Unix® and Linux. During the last 30 years, we have supported at least 40 different platform types and versions. We ported the advertising applications to various flavors of Windows as well (most recently Windows 7 and Windows 8 with touch), but only Layout-8000 has been requested on that platform. Our editorial and digital asset management products are Windows applications, although, even there, the image and story files may be archived on a Linux server.
Why do we remain platform-independent?
Most of the advertising front-end systems in the 1980s were completely bound to specific hardware and operating systems, and they paid the price when new options became available. Can you imagine rewriting an entire application written in DEC assembly language and tuned to a specific DEC operating system? It just didn’t happen! Systems written specifically for Tandem machines, IBM AS400s and HP3000s died out too. And - here’s the lesson in all of this - systems written specifically and exclusively for Windows and Oracle® will someday go too. It’s not that any of these environments were or are intrinsically bad (although some have been better than others); it’s just that they come in and out of favor. Why bind your applications so tightly to any one of them?
How do we stay platform-independent?
In the early history of our company, we did it by using the standard subsets of high-level languages (e.g. Pascal) rather than anybody’s assembly language or non-standard extensions to high-level languages. Then all we had to do was find a compiler for the language on a new platform and, perhaps, translate some control scripts. This seems so obvious now, but there were people who claimed that only with assembly language could you get the performance required for a large multi-user system. We always felt that the right algorithms and database design were much more important for good performance than any tweaking at the machine level, and we have never had performance issues. Over time we became convinced that the best way to keep ourselves platform independent was to create the set of development tools that we use now for all of the applications we build ourselves (i.e., the entire advertising system suite). Only the tools need to be ported to new platforms; every application written with the tools then moves without recoding. Not only platform changes but changes to the look and feel of every application are implemented by enhancements to the tools. Our WIMPS interface (windows, icons, menus, pointers and scroll bars) is now as pretty as anyone’s.
What are these tools?
We call them, collectively, Spice (the Dune books and movie were a big hit when a name was being chosen). They include a screen designer, relational database manager, text editor, composition engine, dialog manager, help manager, application code/formula language, XML processor, report writer and charting package. Back when this was a buzzword, we called Spice a 4th generation1 language. The term is used to refer to non-procedural (or declarative) high-level languages built around database systems. SQL, for example, is a 4th generation language (the first generation was machine language, the second was assembly language and the third included high level procedural languages like Pascal or C).
What is the underlying database?
We use a record manager called C-tree (developed by Faircom, www.faircom.com). We are free to modify the source code, and we have made some enhancements over the years. It provides an ODBC-compliant (Open Data Base Connectivity) interface to our databases, which allows any report writer or query language that is ODBC-compliant to read our Spice databases directly and independently. Crystal Reports® and Visual Basic®, for example, can access Spice’s ODBC-compliant databases. Spice also supports mirroring its databases with others like PostgreSQL. Our Spice report writer (SpiceRAQ) can give you access to all the data in a Spice database - as a printed report, a screen display or an exported text file. (This is in addition to all the standard and user-customizable reports that come with each application.) No one wants to feel that his or her enterprise data - an extremely valuable resource - could ever be held hostage by a vendor who uses a “proprietary” database package. Our customers have never been in this position. We have customers who send data to a “data warehouse” or interface to or from applications from other vendors; sometimes they create these interfaces themselves using the report writer tool and sometimes they ask us to help them do it. Our ODBC-compliant database interface tool makes data access easy for Crystal Reports experts.
What platform(s) do we use ourselves?
Linux is our primary development platform here at SCS. Many of us also have Windows PCs and/or Macs on our desks, but a lot of SCS staff get along just fine with a Linux desktop. We all have access to Linux servers to develop and/or test the SCS applications and to use some of our own in-house databases. We don’t feel the need to go completely “Microsoft-free,” but it would be possible. We have some customers who have made that a goal, and our applications could help them.
Why should you feel comfortable doing business with SCS even if we are “different”?
For one thing, we’ve been serving the newspaper business for a long time (for our company’s history, read “About SCS” on our web site. In fact, we’ve outlasted most of the newspaper vendors that were around when we started, even the industry leaders. I like to think that’s because we have been doing things right. (As a side note, we were founded one year before Oracle and just one year after Microsoft.) We’re not going away anytime soon, and even if we did, we have our application source code in escrow for our customers. We’re committed to the best algorithms that computer science, newspaper expertise and brainpower can provide. We have skilled programmers implementing these algorithms. Customers praise our support staff as the best anywhere. Our long-term technological vision, our dedicated and intelligent staff, our extensive experience in both computers and newspapers and our reliable support are just a few of the reasons to get your newspaper systems from SCS.
Paper checking is not fun. Just ask Bette Norris of the Business Office staff at the Moline (IL) Dispatch. It's a job she's done for years but now, thanks to a new application from Software Consulting Services, LLC of Nazareth PA, her days of manual paper checking are over.
Her new tool is called PaperCheckAdBoss™ or PCAB for short.
The Dispatch is a 40,000 circulation daily that is part of the Small Newspaper Group. Dale Attwood, Production Manager, and Scott Aswege, General Manager, shepherded PCAB from an SCS new product vision to a useful production tool.
Paper checking is a back room process designed to tie the advertising that's published to what's invoiced. Hand measuring and a visual comparison to ad manifests is the usual process. (Those newspapers who abandon paper checking often find their credit managers dealing with unpleasant, labor intensive billing disputes and lack of audit accountability.)
You might think with all the technology of a fully paginated newspaper like the Dispatch, spending over four hours per day measuring and marking off ads wouldn't be necessary. The Dispatch's advertising front-end and A/R is from Brainworks. A DPS system provides production ad management. SCS's Layout-8000™ is used for dummying all products with SCS's InLay™ doing the automatic interfacing to InDesign® which is used for pagination. Believe it or not, PCAB needs to unite all of these systems to do the back-to-front data analysis necessary for effective paper checking.
However, as Bette points out, without paper checking, advertisers find your errors and they typically don't like it when it happens. Bette reports that "We checked the paper both ways during all of April. The only time there was a difference in the result, PaperCheckAdBoss found an error we didn't."
PCAB (which she calls 'AdBoss') has made her paper checking task so much faster and easier. "Whenever there's a problem [with an ad size or placement] it tells me where to go look. I'm more confident that I'm sending out true statements to customers. Since May 1st I have depended on AdBoss 100%."
Prior to PCAB, Bette worked with another business office clerk, checking printed reports and newspaper pages back and forth. Now she can get images of pages as dummied by Layout-8000 and right next to them the pages exported from InDesign for pagination. Or, she can use a simple report which lists exceptions right at the top. A quick glance at the designated exceptions completes the verification. "I can quickly do it by myself," says Bette.
Bette explained that PCAB draws her attention only to ads with a problem, usually on the first run. The problem can be fixed and it never needs to be checked by a person again throughout the run of the ad. She is proud that they "balance to the penny" before bills go out.
In addition, the program produces reports by page and by publication of inches printed, inches scheduled and billed to customers, inches of scheduled in-house ads and inches of fillers added during dummying or pagination. This can be balanced to reports from their ad order entry system to produce better tracking of in-house ads and better figures for non-paid ads.
Scott noted that not only was a formerly 'not fun' job made easy, but there were net labor savings of over 20 hours per week. Dale reported that personnel have already been reassigned to cover work left open by departed employees without hiring new people.
The Business Office saves the checked report from PCAB so that Scott Aswege, the paper's General Manager, can go back at any time and look at it. In the past they kept the marked up papers as an audit trail for two months to handle any advertiser questions. Now they keep only the electronic copies of the PCAB discrepancy reports.
As a beta site customer, Dale worked closely with the SCS developer, Charles Finady, during the installation. Dale describes Charles as "very responsive, very professional and very helpful." [Editor's note: This is what we expect of all of our developers and support staff. Charles is an excellent example.]
A recent email from Dale said, "We just tested the saving function of the newest update from last night and it works PERFECTLY!!! Also adding the lineage and ad count to that report is awesome".
Richard Cichelli, SCS President, says, "It has been a privilege to work with the staff at The Dispatch. These are the kind of customers that make it easy for us to continue writing and installing "smart" software to automate repetitive tasks for newspapers."
Learn about SCS
Published articles that are written by, about, or for SCS are collected here for your viewing.