Skip to content

Detect UI Elements

Description

Uploads a screenshot and runs the element detection on the server and outputs the bounding boxes of the UI elements detected. Element text can optionally be read from the detections with the OCR setting. Typically used with Select box logic to dynamically select mouse click positions

Input Ports

Name dataType sourceType required description
OCR elements boolean manual-only false Option to detect the text inside of the image and output it as text
Screenshot path text link-manual true The path to the image that needs to be detected
Padding number link-manual false Padding will allow you to increase the size of the boxes surrounding the detected text, this increases the accuracy of large text boxes

Scenarios

  • The user wants all the positions of the UI elements inside a image
  • The user wants all the text that exist inside of the image

Use case

Command

Inputs
  • OCR elements: true
  • Screenshot path: D:\example.png
  • Padding: 4
Experienced tips
  • If you do not specify an Screenshot path, the process will stop and provide you with the following error. The "path" argument must be of type string. Received type object
  • If the image could not be found, the process will stop and provide you with the following error. ENOENT: no such file or directory
  • If your boxes are too small inside of the Select from UI elements command, try increasing your Padding
Output

Output

Search Tags

Detect boxes from images Detect text from image