Scanning documents with Tesseract with HOCR

Date last run: 07Dec2019

Introduction

I posted before about using magick and tesseract in postings here and here. Looking again at tesseract because the inputs were changed, I noticed two possibilities to improve the workflow:

• using the tesseract::ocr_data function that returns a data.frame with the scanned text and a confidence rate and a bounding box for this text
• the HOCR argument of the tesseract::ocr function that returns an XHTML document with the same elements as tesseract::ocr_data but with an additional line number

When I used the tesseract::ocr_data, I needed to derive the line number from the bounding box, so I decided to go for the tesseract::ocr(HOCR=T) option. A part of the XHTML document is listed here:

<span class='ocr_line' id='line_1_5' title="bbox 19 227 1087 251; baseline 0 -8; x_size 14.113145; x_descenders 3.1131446; x_ascenders 3.6521802">
      <span class='ocrx_word' id='word_1_28' title='bbox 19 227 147 251; x_wconf 90'>Standaardbereik</span>
      <span class='ocrx_word' id='word_1_29' title='bbox 198 232 242 243; x_wconf 86'>1418</span>

On the internet I found a snippet by Jeroen Ooms that converts the XHTML document to a data.frame. I created a function from the snippet in which I also extracted the line number and made the extraction of the confidence rate conditional. In the previous blogs I used scans to extract the results of medical (blood) tests.

Including cleanup_bw, scan_with_hocr and extract_table in package

The functions used to do the extraction of the medical test results, were generalized for extracting a table from a scan. And being more general they were included in the R-package HOQCutil . Using among others these functions the total workflow then is:

• read an image from file with magick::image_read
  e.g. img1 = magick::image_read('example1.png')
• define the list with cleanup options
  e.g.cln_options1 = list(resize="4000x",trim=10,enhance=TRUE,sharpen=1)
• use the HOQCutil::cleanup_bw function with this list
  e.g. img2 = HOQCutil::cleanup_bw (img1,cln_options1)
• scan (OCR) the cleansed image with HOQCutil::scan_with_hocr
  e.g.df1 = HOQCutil::scan_with_hocr(img2,add_header_cols=F)
• indicate in the columns of df1 which fields belong to the table headers (or alternatively define a headers list)
• extract the table with the HOQCutil::extract_table function
  e.g. df2= HOQCutil::extract_table(df1, headers=NULL,lastline = Inf, desc_above=T) or alternatively
  df2= HOQCutil::extract_table(df1, headers=hdr_desc,lastline = Inf, desc_above=T)

The functions are (of course) described in the help of the HOQCutil package under the name of the function or under scanner_functions. The remainder of this blog shows how this works on an example file.

Example

We will try to scan the table in Figure 1 . We created the .PNG file ourselves (from an MS Excel sheet) so we could give it some ‘features’:

• headers are placed over two lines and contain two words (except last header)
• the last header contains three words
• two descriptions are placed over two lines
• we used the ‘Verdana’ font. With other fonts the scans were far worse!

Read the image from file

filename = 'example.PNG'

img1 = magick::image_read(filename) 

Cleanup the file

The result can be seen in Figure 2 .

cln_options1 = list(resize="4000x",trim=10,enhance=TRUE,sharpen=1) 
img2 = HOQCutil::cleanup_bw(img1,cln_options1) 
img2

OCR the cleansed image

In this step the actual scanning takes place. The results are not too bad but the first and last (is this a coincidence?) row of the table are not correctly scanned. We will show the result (inTable 1) after the next step.

df1  = HOQCutil::scan_with_hocr(img2,add_header_cols = T) 

Prepare for extracting the table

In the previous step we added two columns (header_col and header_col_seq) to the scan results. We use these to indicate which elements will be the header of the table. This could be done interactively by using edit but here we do it by coding:

df1$header_col[c(9,10,11,12,13,14,15,16,17)]= c(1,2,3,4,4,1,2,3,4)

The resulting data.frame can be seen in Table 1:.

Extract the table

The last step is done with the HOQCutil::extract_table function. The resulting table is listed in Table 2. We use desc_above=F because we see in Figure 1 (and Figure 2) that the descriptions are not above the data line. See e.g. the ‘1’ for the first city line and the ‘e 1’ for the last data line.

df2 = HOQCutil::extract_table(df1, headers=NULL, lastline = Inf, desc_above=F)

We also see in Table 2 that the description ‘provinc el’ is not correct. With a perfect scan this would have been ‘provinc e 1’, because descriptions are pasted with a space as separator. So in this case a small correction will have to be made. Also the NA (because no data was present in that cell) should be replaced by whatever the user thinks feasible.
Lesson: always check the results of the scan.

Alternative for the extract

In the previous step we could use the column header_col that we filled in an earlier step with the information concerning headers. However we can also specify this information in list-form:

hdr_desc = list(
  list(c(2,1),c(3,1)),
  list(c(2,2),c(3,2)),
  list(c(2,3),c(3,3)),
  list(c(2,4),c(2,5),c(3,4))
)

df3 = HOQCutil::extract_table(df1,headers=hdr_desc, lastline = Inf, desc_above=T)

In Table 3 we see that this leads to the same headers. We also show the effect of specifying (wrongly) desc_above=T.

Session Info

This document was produced on 07Dec2019 with the following R environment:

  #> R version 3.6.0 (2019-04-26)
  #> Platform: x86_64-w64-mingw32/x64 (64-bit)
  #> Running under: Windows 10 x64 (build 18362)
  #> 
  #> Matrix products: default
  #> 
  #> locale:
  #> [1] LC_COLLATE=English_United States.1252 
  #> [2] LC_CTYPE=English_United States.1252   
  #> [3] LC_MONETARY=English_United States.1252
  #> [4] LC_NUMERIC=C                          
  #> [5] LC_TIME=English_United States.1252    
  #> 
  #> attached base packages:
  #> [1] stats     graphics  grDevices utils     datasets  methods   base     
  #> 
  #> loaded via a namespace (and not attached):
  #>  [1] Rcpp_1.0.3       knitr_1.26       xml2_1.2.2       magrittr_1.5    
  #>  [5] rappdirs_0.3.1   tidyselect_0.2.5 tesseract_4.1    HOQCutil_0.1.15 
  #>  [9] R6_2.4.1         rlang_0.4.2      stringr_1.4.0    highr_0.8       
  #> [13] dplyr_0.8.3      tools_3.6.0      xfun_0.10        ellipsis_0.3.0  
  #> [17] htmltools_0.4.0  digest_0.6.23    assertthat_0.2.1 lifecycle_0.1.0 
  #> [21] tibble_2.1.3     crayon_1.3.4     tidyr_1.0.0      purrr_0.3.3     
  #> [25] captioner_2.2.3  vctrs_0.2.0      zeallot_0.1.0    glue_1.3.1      
  #> [29] evaluate_0.14    rmarkdown_1.18   stringi_1.4.3    pillar_1.4.2    
  #> [33] compiler_3.6.0   backports_1.1.5  magick_2.2       pkgconfig_2.0.3