## Horribly obscure character recognition

So you have a scanned document, and you want to produce a searchable PDF from these images. In my case, I want to digitalize some of my books. There are other tutorials out there, but none of them worked for me. Here's what I did, maybe it helps you. First, get a debian box and install the packages tesseract (this is the OCR software), xsltproc (dark magic), exactimage (for hocr2pdf) and of course pdftk. Now, take your scan and use scantailor to split it into several neat black and white .tif files, one for each page. Now, create a file called fix-hocr.xsl and put this in it:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!-- use on hocr file to fix for hocr2pdf 0.8.9 textbox placement -->
<xsl:template match="/html">
<xsl:text>&#13;</xsl:text>
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<xsl:template match="node()|@*">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<xsl:template match="span[@class='ocr_line']">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
<xsl:element name="br">&#13;</xsl:element>
</xsl:template>
</xsl:stylesheet>

Don't ponder on that. It is a dark conjuring that fixes a bug in hocr2pdf. For every .tif file, we now do the following: * Perform OCR on it, and record the information of where the letters are located in the image. That's what HOCR is all about. * Do some vodoo with fix-hocr.xsl on the HOCR information from tesseract because tesseract's output causes problems with hocr2pdf, at least it did for me. * Use the hocr2pdf tool to create a PDF document which contains two layers: One layer of text information and the original tif image above it. And here's how we do that.
for pg in \$(ls *.tif); do tesseract -l eng -psm 1$pg stdout hocr |
xsltproc -html -nonet -novalid fix-hocr.xsl - |
hocr2pdf -i $pg -o "\${pg%%.tif}.pdf";
done

Ignore the warnings about nonclosing ?xml tags, they're bogus. Now you should have tons of searchable pdf pages, let's merge them into a document.
pdftk *.pdf cat output book.pdf;

The document will be huge. Compressing it is a whole different story of pain and awe. What worked pretty good for me was to convert the PDF to PS and then from PS back to PDF and then from that PDF to DJVU:
pdf2ps book.pdf book.ps;
gs -dCompatibilityLevel=1.4 -dBATCH -dNOPAUSE -dPDFSETTINGS=/ebook -dPDFA=2 -sDEVICE=pdfwrite -sProcessColorModel=DeviceGray -sColorConversionStrategy=/RGB -dUseCIEColor -sPDFACompatibilityPolicy=2 -sOutputFile=book2.pdf book.ps;
pdf2djvu --loss-level=200 --dpi=299 --verbose --monochrome -o book.djvu book2.pdf;

For 300 dpi files. Somehow, setting the DPI to 299 for pdf2djvu shrunk the file size absurdly, as opposed to 300. My wild guess is that 299 somehow allowed pdf2djvu to actually use its lossy compression, while maintaining the 300 did not. As you can see, I have no actual idea what is happening here, so you will probably have to toy with the options a bit like I did. Another side note: The PDF remained huge no matter what I tried, but I got the DJVU down to about 10kb per page, which would be 2mb for 200 pages.

## My Quest for (CPU) Power

This glab post is essentially about running a certain shell command remotely on multiple systems within a network that has been set up for public-key authentication. It's a standard task for experienced system administrators I am sure, but for me it was a little adventure in bash scripting — and I wanted to share it with you. Do you want to know more?