Help with font based image recognition app

Darthed · 09/13/2016, 17:55

basically I don't know what I am supposed to do the code is not really mine so I don't know what I am looking at...I have asked in multiple forums where the problem is and most of the problems so far were either misspelling or old code from older versions of autoit so here it is

Spoiler

HTML Code:

   ;===========================================================================
   ; Settings
   ;===========================================================================
Opt("MustDeclareVars", 1)
Opt("WinTitleMatchMode", 2)
Opt("TrayIconDebug", 1)

   ;===========================================================================
   ; Includes
   ;===========================================================================
#include <character_definitions.au3>

   ;===========================================================================
   ; Setting hotkeys
   ;===========================================================================
HotKeySet("{PAUSE}", "end")

   ;===========================================================================
   ; Testing OCR
   ;===========================================================================
Local $Find, $i, $var, $e, $distx, $disty, $File, $String, $DistanceString, $Line, $Pixel

WinActivate("Paint")

Sleep(1000)

; Colors:
;  White    = 12895428
;  Blue     = 5263532

Local $TimerInit, $TimerDiff
$TimerInit = TimerInit()


$Pixel = 12895428

$File = FileOpen( "output.txt", 2 )

For $Scans = 1 to 2
   ; COORDINATES / PIXEL HERE:
   $Line = GetLines( 61, 43, 318, 139, $Pixel )

   For $LineNum = 1 to $Line[0][0]
      $Find = GetCharacters ( $Line[$LineNum][0], $Line[$LineNum][1], $Line[$LineNum][2], $Line[$LineNum][3], $Pixel )

      Local $String = ""

      Dim $DistanceString[$Find[0][0]+1]

      For $i = 1 to $Find[0][0]
         ; Finds every pixel in the square
         $var = _GetAllPixels($Find[$i][0], $Find[$i][1], $Find[$i][2], $Find[$i][3], $Pixel )

         ; For every value, write the distance from StartX/StartY to
         ; FoundX/FoundY of each pixel
         For $e = 1 to $Var[0][0]
            $distx = $Find[$i][0] - $Var[$e][0]
            $disty = $Find[$i][1] - $Var[$e][1]

            ; Making one big long string for the letter
            $DistanceString[$i] = $DistanceString[$i] & $distx & "," & $disty & ";"
         Next

         ; For some strange reason there was a 0 at the beggining of each string;
         ; this gets rid of it.
         $DistanceString[$i] = StringTrimLeft ( $DistanceString[$i], 5 )

         For $x = 1 to UBound($arCharacters) - 1
            If $arCharacters[$x][1] == $DistanceString[$i] Then
               $String = $String & $arCharacters[$x][0]
            EndIf
         Next

         ;Space stuff, if more than 4 pixels between the end of characters
         ; add a space.

         If UBound($Find) > $i + 1 Then
            ;MsgBox("","", $Find[$i][2] & ", " & $Find[$i+1][0] )
            If Abs($Find[$i][2] - $Find[$i+1][0]) > 4 Then $String = $String & " "
         EndIf

         ;If $i > 1 AND $Find[$i][0]-$Find[$i-1][0] > 10 Then
         ;;; Found a space inbetween letters
         ;   $String =  $String & " "
         ;EndIf

         ;FileWriteLine( $File, $String[$i] & "=" & $DistanceString[$i])
         ;MsgBox("","", $String )
      Next

      FileWriteLine( $File, $String )
   Next

   $Pixel = 5263532
Next

FileClose($File)

$TimerDiff = TimerDiff ($TimerInit)
MsgBox("", "", $TimerDiff)

   ;===========================================================================
   ; Functions
   ;===========================================================================
Func _GetAllPixels($StartX, $StartY, $EndX, $EndY, $Color)
   Local $aCoord, $Length, $Width, $Area, _
         $i, $x, $y, $CurrentElement = 1

   ; Get area of rectangle to initiate array
   $Length = Abs($StartX - $EndX)
   $Width = Abs($StartY - $EndY)
   $Area = $Length * $Width
   Dim $aReturn[$Area + 1][2]

   ; Search inside rectangle for the pixel
   For $x = $StartX to $EndX
      For $y = $StartY to $EndY
         If PixelGetColor( $x, $y) = $Color Then
            $aReturn[$CurrentElement][0] = $x
            $aReturn[$CurrentElement][1] = $y
            $CurrentElement = $CurrentElement + 1
         EndIf
      Next
   Next

   ReDim $aReturn[$CurrentElement][2]
   $aReturn[0][0] = $CurrentElement - 1

   Return $aReturn
EndFunc

Func GetLines($StartX, $StartY, $EndX, $EndY, $Color)
   Local $arLineCheck = 0, $intElement = 1, $FirstFound = 0
   Dim $arNewLine[255][4]

   #CS
   DEBUG
   $arStartCoord[0] =

   $arStartCoord = PixelSearch ($StartX, $StartY, $EndX, $EndY, $Color)
   MsgBox("","", $arStartCoord[0] & ", " & $arStartCoord[1])
   If @error = 1 Then
      SetError(1)
      Return
   EndIf

   $arNewLine[$intElement][0] = $arStartCoord[0]
   $arNewLine[$intElement][1] = $arStartCoord[1]
   #CE


   While $StartY < $EndY
      ;Scan 1 pixel line

      ;debug
      ;MouseMove( $StartX, $StartY, 0)
      ;Sleep(500)
      ;MouseMove( $StartX, $StartY+100, 0)
      $arLineCheck = PixelSearch ($StartX, $StartY, $EndX, $StartY, $Color)

      ; Find first line
      If @error = 0 AND $FirstFound = 0 Then
         $arNewLine[$intElement][0] = $StartX
         $arNewLine[$intElement][1] = $arLineCheck[1]

         $FirstFound = 1
      EndIf

      ; If nothing found, set end of line
      If @error = 1 AND $FirstFound > 0 Then
         $arNewLine[$intElement][2] = $EndX
         $arNewLine[$intElement][3] = $StartY

         $intElement = $intElement + 1;

         $FirstFound = 0
         #CS
         ; Looking for another line within 15 pixels
         $arLineCheck = PixelSearch ($StartX, $StartY, $EndX, $StartY+15, $Color)

         ; If something is found, add it
         If @error = 0 Then
            ; Reset the new Y coordinate start point for line search
            $arLineCheck[1] = $StartY

            $arNewLine[$intElement][0] = $arLineCheck[0]
            $arNewLine[$intElement][1] = $arLineCheck[1]
         EndIf
         #CE
      EndIf

      $StartY = $StartY + 1;
   WEnd

   $arNewLine[0][0] = $intElement - 1
   ReDim $arNewLine[$intElement][4]

   Return $arNewLine
EndFunc

Func GetCharacters($StartX, $StartY, $EndX, $EndY, $Color, $Height = 12, $Width = 12)
   Local $vSearchCoord, $vCharacter = 2, $vLineCheck, $FirstFound = 1, $Check = 0
   Local $Error
   Dim   $vCharCheck, $aNewCharacter[255][4]

   $vSearchCoord = PixelSearch ($StartX, $StartY, $EndX, $EndY, $Color)

   If @error = 1 Then
      SetError(1)
      Return
   EndIf

   $aNewCharacter[1][0] = $vSearchCoord[0]
   $aNewCharacter[1][1] = $vSearchCoord[1]

   ;Check for a line between characters to differ between them
   While $vSearchCoord[0] < $EndX
      $vSearchCoord[0] = $vSearchCoord[0] + 1
      $vLineCheck = PixelSearch ($vSearchCoord[0], $StartY, $vSearchCoord[0]+1, $EndY, $Color)

      $Error = @error
      #CS
      ; Check for another character can be interrupted by a gap in pixels in a character
      ; so I check for TWO pixels between each character
      ; I also add a check for spaces, by proving if there's more than 1 space between, then there's a pixel
      #CE
      If $Error = 0 AND $FirstFound = 0 Then
         ; Reset the new X coordinate start point for line search
         $vSearchCoord[0] = $vLineCheck[0]

         $aNewCharacter[$vCharacter][0] = $vLineCheck[0]
         $aNewCharacter[$vCharacter][1] = $vLineCheck[1]

         $vCharacter = $vCharacter + 1

         $FirstFound = 1

         $Check = 0
      EndIf

      If $Error = 1 Then
         $Check = $Check + 1
      EndIf

      If $Error = 1 AND $Check = 1  Then
         ; Nothing found:
         ;     Adding end of character coord
         $aNewCharacter[$vCharacter-1][2] = $vSearchCoord[0]
         $aNewCharacter[$vCharacter-1][3] = $EndY

         $FirstFound = 0
      EndIf
   WEnd

   $aNewCharacter[0][0] = $vCharacter - 1
   ReDim $aNewCharacter[$vCharacter][4]

   Return $aNewCharacter
EndFunc

Func End()
   Exit
EndFunc

elmarcia · 09/13/2016, 22:44

Worst ocr ever seen, is this s*** working?
Maybe you should make it work another way or find any project with ocr and try from there.
Useful links:

Shadow992 · 09/14/2016, 00:22

Quote:

Originally Posted by elmarcia

Worst ocr ever seen, is this s*** working?
Maybe you should make it work another way or find any project with ocr and try from there.
Useful links:

I see you are not having any kind of experience programming OCR and similar computer vision tasks.
Even if this OCR is not perfect, it is still usable in certain cases. This code uses some kind of k nearest neighbour classifier, which has been shown in literature that this classifier works best among all others (SVMs, NNs, RandomForest, ...) if you know the exact font (like you mostly do when you are working with computer generated texts). Even if you do not know the used font it may still work as good as using convolutional neural networks.

@Darthed
It highly depends on the application you want to develop (like always in computer vision). There is no solution which works for everything and every case. This kind of general purpose AI is decades of research away from us. So you should specifiy your goals, show some example texts you want to recognize and tell us what you know.
In general this is no easy task which you can solve in 1-2 weeks if you have basically no knowledge in the computer vision subject. Therefore I suggest to use Tesseract this is some kind of general purpose OCR but of coz it is not working perfectly in every case.

Darthed · 09/15/2016, 13:47

Thanks for the replys
this is the github page of the author by the way

I did modify it a little bit to stop it from giving errors but no luck with actually reading anything.. I think what it's supposed to do is detect text on the screen and then output it into output.txt and close the program but when I open it and hover over the text with the mouse it propts a random string of numbers and an Ok button...

Shadow992 · 09/15/2016, 13:58

Quote:

Originally Posted by Darthed

Thanks for the replys
this is the github page of the author by the way I did modify it a little bit to stop it from giving errors but no luck with actually reading anything.. I think what it's supposed to do is detect text on the screen and then output it into output.txt and close the program but when I open it and hover over the text with the mouse it propts a random string of numbers and an Ok button...

Because you do not use the same font as the author did, also the splitting of
characters may need to be done on another way. As already mentioned, if you want to use it for something different than the author did, you have to change many things. Maybe that much things that recoding it is eveneasier/better.
But because your knowledge in OCR is very limited, vour best way to go is using Tesseract or some similar ready to use OCRs. They will not work perfectly, too. But at least they will work better than this approach you presented.

Edit:
If you would provide some screenshots we may be able to help you more, otherwise this is a guessing game...

Darthed · 09/15/2016, 15:01

Quote:

Originally Posted by Shadow992

Because you do not use the same font as the author did, also the splitting of
characters may need to be done on another way. As already mentioned, if you want to use it for something different than the author did, you have to change many things. Maybe that much things that recoding it is eveneasier/better.
But because your knowledge in OCR is very limited, vour best way to go is using Tesseract or some similar ready to use OCRs. They will not work perfectly, too. But at least they will work better than this approach you presented.

Edit:
If you would provide some screenshots we may be able to help you more, otherwise this is a guessing game...

screenshots ? as in like... this one? since it waits for "WinActivate("Paint")" ....

Spoiler

Shadow992 · 09/15/2016, 15:22

Quote:

Originally Posted by Darthed

screenshots ? as in like... this one? since it waits for "WinActivate("Paint")" ....

Spoiler

This is something we can work with. At least now we know that you want to use it for Diablo 2.
Your script obviously does not find lines and/or characters to recognize. This can be due to invalid startX/startY coordinates or due to changed fonts, ...

Especially this looks quite "wrong" for me. I guess you did not change these coords:

PHP Code:


			
; COORDINATES / PIXEL HERE:
$Line = GetLines( 61, 43, 318, 139, $Pixel )

If you changed this and it still does not find any lines/characters, you have to debug it by yourself. The messagebox you see is this msgbox:

PHP Code:


			
MsgBox("", "", $TimerDiff)

So it basically only shows you how long this script ran (~3800ms = ~4s). However as you are not very used to programming, you should search for another solution or for someone who gonna code something for you. Maybe you are able to find someone on this board but I guess no.

Darthed · 10/06/2016, 17:03

Yeah thanks for the reply (yeah been busy for long time) now I actually know what the those actually lines do... I did try the app by changing the coordinates in $Line = GetLines part but it didn't really work out well it started running for way longer time(I basically put it on my default screen resolution) so the image search went completely slow mode and even if I narrowed down the coordinates to search for it was still not able to detect the text it might be because of my fonts but I am not sure if the reason is because my font in Diablo 2 is changed because I did try out custom fonts I was not sure if I successfully uninstalled it or not otherways this should definitely work with default diablo 2 fonts...

Also I am not exactly sure how to interpret the screen coordinates does this change work like this ??

Code:

$Line = GetLines( X, Y, X, Y, $Pixel )

then it should probably be like ( 0, 768, 1366,766) maybe ??

So I would really appreciate if somebody could test this script out see what is wrong with it or try to make a new similar one I really wanted to make sort of automatic cube recipe app where it would do certain actions based on the stats of an item... the hardest part I couldn't manage for me was making it speedy I wanted it to detect the characters on the screen as fast as possible and just get on with the rest of the functions...
So if somebody knowledgeable or with OCR experience revisits this it would be really awesome!