marcincichocki / breach-protocol-autosolver Goto Github PK

• @/core => core functionality, sequence, solving, ocr. Platform agnostic. Move registry here
• @/common => common functionality, utils, helper fns, shared code
• @/common/node => nodejs shared code
• @/common/electron => shared electron code
• @/electron/main => main process
• @/electron/renderer => react
• @/electron/worker => worker

importing from @/core in renderer is problematic since its webpack configuration is not handling externals at all.
Besides, core should be neutral and have no node deps.

In some cases forms will be disabled(flag or status dependent).

TODO:

• check if cyberpunk ui have disabled controls(rtx?)

Icon will be used in electron setup, electron bin and tray.

This breach protocol broke solver. Investigate what is going on.

If user start ocr not on breach protocol screen or ocr could not resolve symbols, we should skip current job.

ocr validatation could work like this:

const isGridValid = grid.every(s => ALLOWED_SYMBOLS.include(s));

Describe the bug
When using language like chinese or japanese(or other with custom font) ocr report errors.

To Reproduce
Set interface language with custom font and then try to use autosolver.

Expected behavior
Should work like it does with english interface.

Software information

• OS: Windows 10
• autosolver version: 1.4.0
• autosolver flags(if applicable):
• monitor resolution: 2560x1440
• gamma(if applicable):

Cyberpunk 2077 information

• version: 1.2
• resolution: 2560x1440
• language: see above
• gamma(if applicable):

Additional context

Some kind of flexbox issue

remove old TODOS, refactor game, etc

remove node client, remove build from actions, remove build script

There are few edge cases that ocr testing doesn't cover now:

• high/low gamma(#53)
• bufferSize 9
• ultrawide
• 4:3 low resolution

bpa should support multiple platforms because Cyberpunk 2077 can be played on linux/mac through proton/wine/stadia/geforce now.

TODO:

• try upstream version on gforce now.
  UPDATE: Recognition is working. Robot clicks in incorrect places(prob gnow issue with resolution). Tried scaling on/off, neighter works.
  UPDATE 2: nircmd does not work well with gnow. Mouse moves into wrong places, and keyboard does not work at all. Possible solution is to implement different bin than nircmd(autohotkey?)

• see if ocr works through compression
  UPDATE works, might require fixed threshold.

• see if outputs get send to stadia/gforce now
  UPDATE clicks are send
  UPDATE 2 keystrokes are NOT

• see if stadia/gforce now support fullscreen and how it affects robot clicks UPDATE only tried gnow in app, and robot clicks in incorrect places.

• add required native deps for linux/mac and update robot(xdotool and imagemagik on linux, use sudo-propmt)

• update docs on required deps(imagemagik for linux)

• research multiplatform release and builds

Buffer fragment uses brute force approach for its recognition. Since threshold changes image from white to black, image bits are kinda sorted. This means we can use binary search for significant boost in performance(worst case O(7) since even lowest gamma don't force threshold lower than 128).

UI needs custom form controls:

• select
• range slider
• switch

• inactive daemons always highlight squares
• not used buffer keep square highlight
• not used buffer have incorrect styles
• history list change text on click
• app throws when there are no items in history
• cannot read status undefined(when last item in history is selected and new bp is solved). This is because we only keep N elements in store.

In new electron client, history feature will keep track of all attempted BPs. Source (png or jpg) can weight up to few MB which is unacceptable, therefore it's better to remove all sources for BPs that succeeded(failures are usefull for bug reports).

Sometimes, it's good to keep them(debug mostly) so it's useful to put it behind flag.

To display BPs in history viewer is required which will display lots of data in compact and clear way.

• grid
• daemons
• bufferSize
• resolved sequence
• used parts
• path
• exit strategy

preview:

Describe the bug
Autosolver broke after successful ocr read

To Reproduce
Try to solve attached png

Expected behavior
Should solve it.

Software information

• OS: Windows 10
• autosolver version: 1.4.0
• autosolver flags(if applicable):
• monitor resolution: 2560x1440
• gamma(if applicable):

Cyberpunk 2077 information

• version: 1.2
• resolution: 2560x1440
• language: japanese
• gamma(if applicable):

Additional context

EDIT: looks like bottleneck is sequence production(not enough fallback values). Also lack of error handling makes it weird for end user.

If worker has not finished loading, and user goes into settings, error will be thrown, since activeDisplayId is not yet set(first launch).

Furthermore updating settings outside of settings page can lead to incorrect values in form controls

ideas:

• block settings until worker is ready
• listen for worker ready event and update form state
• show spinner util worker is ready

There are no tests for OCR which is vital part of this project. We should ensure that most common resolutions are supported:

• 1080p
• 1440p
• 4k

That will propably require adjusting ./src/config.json for each resolution. In worst case it would require retraning tesseract.

Day 1 notes:

• 1080p, 1440p and 4k mostly work on existing code, small improvments necessary
• grid and daemons OCR works very well, no issues there(some edge cases on 4k since res is so high, tesseract recognize some leftover junk, but it's mostly fixed by separate threshold for each res)
• buffer OCR is a mess, It will require separate .traineddata because it's so bad(it's a miracle that it worked so well anyway)
  

Day 2 notes:

• buffer ocr scrapped, reimplemented by counting raw pixel data(size of buffer scales with resolution). Ocr might be used later on as fallback
• much more reliable than ocr, but requires very specific threshold
• still need to check impact of gamma correction on this method

It would be great idea to create debug mode. In that mode all screenshots would be saved along with ocr data. That would be invaluable for issue reproduction.

Stuff to do/research:

• can nodejs write to disk like that
• what to do about permissions(say user puts bin into "C:\Program Files")
• what about size(4k images are big, that would quickly grow(--max-size? --debug-limit?)
• create issue template that would work well with debug data
• extract data from debug mode into test data(if ocr is to be trusted lol)

Describe the bug
Max buffer size is 9 not 8(base 8 from cyberdeck + 1 from inteligence passive lv19). This cause every breach protocol to fail.

To Reproduce
Buy cyberdeck with 8 buffer and get level 19 in inteligence, than try breach anything

Expected behavior
Should work

Software information

• OS:
• autosolver version: 1.6.1
• autosolver flags(if applicable):
• monitor resolution:
• gamma(if applicable):

Cyberpunk 2077 information

• version:
• resolution:
• language:
• gamma(if applicable):

Additional context

https://github.com/actions/create-release
https://github.com/actions/upload-release-asset

History feature in electron app will require schema and migration configuration for future releases.

interface HistoryEntry {
  id: string; // uuid
  source: string | null; // path to source image(png or jpg) in `userData`. `null` if job succeded, unless flag is specified
  dateStart: string; // when job has started
  dateEnd: string; // when job has ended
  options: ?; // with what options job was started
  ocr: ?; // serialized BreachProtocolRecognitionResult map. *MUST* include base64 8bit fragment images. Should contain threshold?
  sequences: Sequence[]; // list of produced sequences. Need toJSON method
  game: BreachProtocol; // ? is this required?
  result: BreachProtocolResult;
  exitStrategy: ExitStrategy;
  status: boolean; // ok/err
}

Describe the bug
After each usage of autosolver(successful) memory usage goes up, and doesn't decrease over time.

To Reproduce
Make screenshot of breach protocol, then open task manager and set it to be always on top, then open screenshot on full screen and run autosolver. Observe memory usage go up.

Expected behavior
Memory should be released after each run.

Software information

• OS: Windows 10
• autosolver version: 1.4.0
• autosolver flags(if applicable):
• monitor resolution:
• gamma(if applicable):

Cyberpunk 2077 information

• version:
• resolution:
• language:
• gamma(if applicable):

Additional context

EDIT: Couldn't locate leak by node built it inspector, memory usage is stable while running from a script. Next attempt will require hooking up devtools into nexe binary somehow, this might be challenging since nexe won't build binary with tesseract debug flag, and that is required to launch devtools in the first place.

With removal of node client these dependencies are no longer required:

• @octokit/rest
• commander
• inquirer
• iohook
• open
• ora
• sound-play

and these devDependencies:

• @types/copy-webpack-plugin
• @types/inquirer
• concurrently
• copy-webpack-plugin
• nexe
• nodemon
• ts-node
• tsc-alias
• upx(?)
• zip-a-folder

query github api and open link to asset if user said so(add inquirer question)

https://docs.github.com/en/rest/reference/repos#get-the-latest-release
https://github.com/sindresorhus/open

kill process afterwards

Using node binary as client has become way to much of a burden, withconstant updates, new features, flags etc. Creating new client in electron would give user simpler shortcuts, auto updates, explicit calibration and more.

It would be great if sound could be played when screenshot is taken to notify user about ocr success or error.

todo:

• research nodejs audio libs
• find some free sound library
• implement #18

Clicking on key bind input, entering some combination and clicking outside cause incorrect keycombination to appear.

Either restore last combination or save current.

Is your feature request related to a problem? Please describe.
Clicking takes too much time, there is no point slowing computer down.

Describe the solution you'd like
boost delay from default 75ms to something faster.

Describe alternatives you've considered
none

Additional context

Describe the bug
If unused daemons overlap with sequence and there is space in the buffer, BP will not exit automatically.

To Reproduce
Use attached screenshot

Expected behavior
Leftover daemons overlap should be taken into account.

Software information

• OS:
• autosolver version:
• autosolver flags(if applicable): --experimental-buffer-size-recognition
• monitor resolution:
• gamma(if applicable):

Cyberpunk 2077 information

• version:
• resolution:
• language:
• gamma(if applicable):

Additional context

Custom form control use defaultValue prop, refactor to use value

Some dependencies cannot be included in the package like vc_redist.x64.exe, it should be noted in in readme or FAQ,

wilix-team/iohook#193

desktop-screenshot have multi monitor support. Therefore it would be nice if we could support it as well. preferably by a flag like: --use-monitor n

Note: moving flag thing into separate issue, because there are a lot of flags that could potentially be implemented, and that overlap nicely with switch to commander. For now, every time user launches autosolver it will inqueire about which monitor to use if there is more than one.

Stuff that CI/CD could help with:

• run tests for PR
• create releases on new tags and upload binaries

Some daemons or more valuable than others, simple index order might not be sufficient. Also, there are cases in which there are 2 sequences with same amount of parts and same length. Sequence that will be chosen is kind of random(alphabetic order) which might not represent "weight" of sequence. For example, sequence with daemons of indexes 0,1 might be worse that 1,2. Ideal solution might require OCRing daemon type, which in turn require language recognition(might be slow).

• history
• settings
• status bar

Currently --debug-limit flag and debug functionality in general only work when there is no OCR error which is counterintuitive.

Is your feature request related to a problem? Please describe.
When sequence length is shorter than available buffer, breach protocol doesn't exit automatically.

Describe the solution you'd like
Mini game should close when:

• there is no more buffer
• all daemons have been completed
• sequence doesn't contain every daemon, and there is space in buffer still

Describe alternatives you've considered
clicking on exit button or pressing esc is annoying

Additional context

When user have n items in history, and changes historySize option to number that is lower than n than extra items will not be deleted.

• parse cli flags in main process
• merge default, json and flags and set it as initial state
• feed it to worker and renderer

Flags that have to be changed in some way:

• --key-bind electron is using different format
• --lang (?) that would require i18n in react
• --skip-update-check(?)

Electron client settings:

• width/height set by user
• position(?)
• selected monitor(shouldn't it be in flags?)
• list of monitiors