¶ OCR receipt
The AI reads images of receipts and converts them into text.
Beta version: The platform is in a development stage of the final version, which may be less stable than usual. The efficiency of platform access and usage might be limited. For example, the platform might crash, some features might not work properly, or some data might be lost.
¶ Training model
¶ Model
OCR receipt model consists of two components: OCR Engine and Receipt parser. Each component is working independently. First, the input of this OCR receipt model is a receipt image, this image is then feed to OCR engine model to transcribe texts and bounding boxes. Then, the bounding boxes and text are used as an input to receipt parser to predict necessary values from the transcribed text.
- OCR Engine: The OCR Engine extracts the text as well as bounding box coordinates from text. This can be seen as an image-to-text model where the model transcribes any text present on the image. For OCR receipt task, these text and bounding boxes are then used to further extract relevant information.
- Receipt Parser: Receipt Parser model receives an input of bounding boxes and corresponded text from the OCR Engine. The model identifies any corresponded value based on the given list of bounding boxes such as Tax ID, Grand Total, Date, etc. Heuristically, the model learns the relationship between neighbor boxes and predict its corresponded value accordingly. For example, the grand total value is mostly correlated with keyword “Grand Total”, “Balance Due”, “Total Inc VAT”, etc.
¶ Model Constraint
Although the system can read a clear and perfectly aligned receipt without any issue, in some condition, the model could fail to predict receipt fields. In other words, the model capability is limited to the image that are clearly readable by human. If the receipt images are hard to read for human, those images are very likely to resulted in an inaccurate prediction. Additionally, the current version only supports only four fields: grand total, date, time, shop tax id, and receipt id.
¶ Model Evaluation
The model is evaluated using accuracy metric, where the metric measures the correctness of the prediction for each field. For each field, a prediction is counted as correct ONLY IF the prediction and label in that specific field are EXACTLY matched.
¶ Using Approach
¶ Input
The input image can be sent in a form of multi-part form data using the key name “files”. Here’s an example python snippet for requesting the API
import requests
def predict(url: str, img_path: str) -> dict:
files= [
('files', (os.path,basename(img_path), open(img_path, 'rb'), 'image/jpeg'))
]
response = requests.post(url, files,files)
return response.json()
¶ Tips on using the model
When using the receipt parsing model, we recommend integrators to check whether:
- The receipt is clear, and text is clearly visible.
- Only one receipt presence in an image.
- Receipt images are in correct orientation.
If these conditions are applied, we can ensure that the model can perform best since it’s within a desired environment for the model.
¶ Output
The output is a structured JSON containing 6 main parts:
- Shop_tax_id: The extracted shop tax ID from the receipt. Return null if not found.
- Grand_total: The grand total of the receipt purchase with VAT included.
- Receipt_id: A unique identifier of the receipt.
- Receipt_date: A raw extracted date string representation from the receipt
- Receipt_time: A raw extracted time string representation from the receipt
- Receipt_timestamp: A formatted date time in ISO format
The example of output is shown below.
[
{
"shop_tax_id": "0101111000111",
"grand_total": "1234.0",
"receipt_id": "E000000000A1234,
"receipt_date": "31-12-2566",
"receipt_time": "19:00",
"receipt_timestamp": "2023-12-31T19:00:00+00:00"
}
]