Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Image recoginiton and clicking on the screen

i need python libraries to make a bot that will recognize an image of the button and click on it, button’s position is randomized every click.
it should collect the image and click by itself, without user interaction

i have used OpenCV in the past, but i do not know how to automate the clicking on the screen

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

To automate clicking on the screen based on the recognition of a button in an image, you can use the pyautogui library to simulate mouse clicks. Here’s a simple example using Python, OpenCV for image recognition, and pyautogui for automating the mouse clicks:

import cv2
import pyautogui
import numpy as np
import time

# Function to find the position of the button in the screenshot
def find_button(screen, button_template):
    result = cv2.matchTemplate(screen, button_template, cv2.TM_CCOEFF_NORMED)
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
    return max_loc

# Function to simulate a mouse click at the specified position
def click_button(position):
    pyautogui.click(position[0], position[1])

# Path to the screenshot of the screen
screenshot_path = 'path/to/screenshot.png'

# Path to the template image of the button
button_template_path = 'path/to/button_template.png'

# Load the images
screenshot = cv2.imread(screenshot_path)
button_template = cv2.imread(button_template_path)

# Find the button in the screenshot
button_position = find_button(screenshot, button_template)

# Click the button
click_button(button_position)
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading