microsoft / form-recognizer-toolkit

Form Recognizer Toolkit (FRTK) provides a set of components and features to accelerate development based on Form Recognizer service.

Project

This repo has been populated by an initial template to help get you started. Please make sure to update the content to build a great experience for community-building.

As the maintainer of this project, please make a few updates:

• Improving this README.MD file to provide a great experience
• Updating SUPPORT.MD with content about this project's support experience
• Understanding the security reporting process in SECURITY.MD
• Remove this section from the README

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

form-recognizer-toolkit's Issues

ServiceRequestError: No connection adapters were found

Hi,

I'm trying to run the /SampleCode/Python/sample_figure_understanding.ipynb file, but when it tries to call the Azure Document AI REST API, it throws the following error.

ServiceRequestError: No connection adapters were found for '"https://xxxxxxx-westus2.cognitiveservices.azure.com/" /documentModels/prebuilt-layout:analyze?api-version=2024-02-29-preview&outputContentFormat=markdown# Your Azure Document Intelligence resource's endpoint value./documentintelligence'

I've tried deploy the Document AI service in EastUS and West US2, same issue.
I've also tried to execute sample API call as described in the REST API 2024-02-29-preview documentation, same issue.

any advise ?

thanks in advance,

regards,
Alex

Unable to pass the file_path

I can't pass documents from my local desktop and unable to provide Url or file_path

Compile fails "Compiled with problems: / Module build failed / TypeError: Cannot read properties of undefined (reading 'replace')

Compiled with problems:

ERROR in ./src/index.scss (./node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[1]!./node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[2]!./node_modules/resolve-url-loader/index.js??ruleSet[1].rules[1].oneOf[7].use[3]!./node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[4]!./src/index.scss)

Module Error (from ./node_modules/sass-loader/dist/cjs.js):
Node Sass does not yet support your current environment: Windows 64-bit with Unsupported runtime (108)
For more information on which environments are supported please see:
https://github.com/sass/node-sass/releases/tag/v7.0.3

ERROR in ./src/index.scss (./node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[1]!./node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[2]!./node_modules/resolve-url-loader/index.js??ruleSet[1].rules[1].oneOf[7].use[3]!./node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[1].oneOf[7].use[4]!./src/index.scss)

Module build failed (from ./node_modules/resolve-url-loader/index.js):
TypeError: Cannot read properties of undefined (reading 'replace')

Failed to access local files

ERROR in src/view/components/messageModal/messageModal.tsx:39:17

TS2769: No overload matches this call.
  Overload 1 of 2, '(props: IBaseModalProps | Readonly<IBaseModalProps>): Modal', gave the following error.
    Type '{ isOpen: boolean; isLoading: boolean | undefined; header: Element; body: ReactNode; footer: Element; buttonGroup: Element; loadingText: string | undefined; width: number | undefined; onClose: (() => void) | undefined; onDismiss: (() => void) | undefined; }' is not assignable to type 'IntrinsicAttributes & IntrinsicClassAttributes<Modal> & Readonly<IBaseModalProps> & Readonly<...>'.
      Property 'isOpen' does not exist on type 'IntrinsicAttributes & IntrinsicClassAttributes<Modal> & Readonly<IBaseModalProps> & Readonly<...>'.
  Overload 2 of 2, '(props: IBaseModalProps, context: any): Modal', gave the following error.
    Type '{ isOpen: boolean; isLoading: boolean | undefined; header: Element; body: ReactNode; footer: Element; buttonGroup: Element; loadingText: string | undefined; width: number | undefined; onClose: (() => void) | undefined; onDismiss: (() => void) | undefined; }' is not assignable to type 'IntrinsicAttributes & IntrinsicClassAttributes<Modal> & Readonly<IBaseModalProps> & Readonly<...>'.
      Property 'isOpen' does not exist on type 'IntrinsicAttributes & IntrinsicClassAttributes<Modal> & Readonly<IBaseModalProps> & Readonly<...>'.
    37 |         return (
    38 |             <Modal
  > 39 |                 isOpen={isOpen}

....

[1]
[1] ERROR in src/view/containers/labelPane/labelPane.tsx:3:61
[1] TS2305: Module '"@fluentui/react"' has no exported member 'Text'.
[1]     1 | import * as React from "react";
[1]     2 | import { connect, ConnectedProps } from "react-redux";
[1]   > 3 | import { Stack, IconButton, DirectionalHint, IButtonStyles, Text } from "@fluentui/react";
[1]       |                                                             ^^^^
[1]     4 |
[1]     5 | import { ApplicationState } from "store";
[1]     6 | import {
[1]

windows can run this?

sample_rag_langchain.ipynb : "message": "The file is corrupted or format is unsupported. Refer to documentation for the list of supported formats."

Hello,

I'm facing an issue :
I first used the sample_figure_understanding.ipynb notebook and get markdown files generated.
Then I used the sample_rag_langchain.ipynb to split and index my md file into an Azure Search, but facing the following issue :

---

HttpResponseError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_25944\1031862820.py in ?()
22
23 # Initiate Azure AI Document Intelligence to load the document
24 loader = AzureAIDocumentIntelligenceLoader(file_path=file_path, api_key = doc_intelligence_key, api_endpoint = doc_intelligence_endpoint, api_model="prebuilt-layout")
25
---> 26 docs = loader.load()
27
28 # Assuming each file contains a single document for simplicity
29 docs_string = docs[0].page_content

c:\Python311\Lib\site-packages\langchain_core\document_loaders\base.py in ?(self)
28 def load(self) -> List[Document]:
29 """Load data into Document objects."""
---> 30 return list(self.lazy_load())

c:\Python311\Lib\site-packages\langchain_community\document_loaders\doc_intelligence.py in ?(self)
92 ) -> Iterator[Document]:
93 """Lazy load given path as pages."""
94 if self.file_path is not None:
95 blob = Blob.from_path(self.file_path) # type: ignore[attr-defined]
---> 96 yield from self.parser.parse(blob)
97 else:
98 yield from self.parser.parse_url(self.url_path) # type: ignore[arg-type]

c:\Python311\Lib\site-packages\langchain_core\document_loaders\base.py in ?(self, blob)
122
123 Returns:
124 List of documents
125 """
--> 126 return list(self.lazy_parse(blob))

c:\Python311\Lib\site-packages\langchain_community\document_loaders\parsers\doc_intelligence.py in ?(self, blob)
76 def lazy_parse(self, blob: Blob) -> Iterator[Document]:
77 """Lazily parse the blob."""
78
79 with blob.as_bytes_io() as file_obj:
---> 80 poller = self.client.begin_analyze_document(
81 self.api_model,
82 file_obj,
83 content_type="application/octet-stream",

c:\Python311\Lib\site-packages\azure\core\tracing\decorator.py in ?(*args, **kwargs)
74 passed_in_parent = kwargs.pop("parent_span", None)
75
76 span_impl_type = settings.tracing_implementation()
77 if span_impl_type is None:
---> 78 return func(*args, **kwargs)
79
80 # Merge span is parameter is set, but only if no explicit parent are passed
81 if merge_span and not passed_in_parent:

c:\Python311\Lib\site-packages\azure\ai\documentintelligence_operations_operations.py in ?(self, model_id, analyze_request, pages, locale, string_index_type, features, query_fields, output_content_format, **kwargs)
3623 polling: Union[bool, PollingMethod] = kwargs.pop("polling", True)
3624 lro_delay = kwargs.pop("polling_interval", self._config.polling_interval)
3625 cont_token: Optional[str] = kwargs.pop("continuation_token", None)
3626 if cont_token is None:
-> 3627 raw_result = self._analyze_document_initial( # type: ignore
3628 model_id=model_id,
3629 analyze_request=analyze_request,
3630 pages=pages,

c:\Python311\Lib\site-packages\azure\ai\documentintelligence_operations_operations.py in ?(self, model_id, analyze_request, pages, locale, string_index_type, features, query_fields, output_content_format, **kwargs)
514 if _stream:
515 response.read() # Load the body in memory and close the socket
516 map_error(status_code=response.status_code, response=response, error_map=error_map)
517 error = _deserialize(_models.ErrorResponse, response.json())
--> 518 raise HttpResponseError(response=response, model=error)
519
520 response_headers = {}
521 response_headers["Retry-After"] = self._deserialize("int", response.headers.get("Retry-After"))

HttpResponseError: (InvalidRequest) Invalid request.
Code: InvalidRequest
Message: Invalid request.
Inner error: {
"code": "InvalidContent",
"message": "The file is corrupted or format is unsupported. Refer to documentation for the list of supported formats."
}

When I look the markdown file generated, I can see that titles (#) are represented by "==="

(I tried to manually make the change, but still facing the same issue, can anybody help ?

readBinary should request an arraybuffer?

Hi there, in Client/src/providers/storageProvider, should the readBinary function request an arraybuffer as the response? I think the axios default is a json, and this causes problems when reading a pdf for example.

    public async readBinary(filename: string, ignoreNotFound?: boolean): Promise<Buffer | undefined> {
        try {
            const api = `${serverUrl}/files/${filename}`;
            const result = await getWithAutoRetry(api, {
                responseType:"arraybuffer" //Added this
            }
        );
            return result.data;
        } catch (exception) {
            this.storageErrorHandler(exception, ignoreNotFound);
        }
    }

Unable to see hover popup on Labeling UX

I've downloaded the latest version as of now on my computer and followed instructions of README.MD. When I run the project, am not able to see any popup when hovered over PDF file for which OCR is already done. Instead, it simply shows yellow boxes.

Can someone please help to fix this?

Client installation fails with Visual Studio 2022 CE and Python 3.10

When trying to run npm i for the client, the installation routine stops:

npm ERR! gyp info find VS "C:\Program Files\Microsoft Visual Studio\2022\Community"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Program Files\Python310\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args 'C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'msvs',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build\config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp\addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\USERNAME\AppData\Local\node-gyp\Cache\18.12.1\include\node\common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args '-Dnode_root_dir=C:\Users\USERNAME\AppData\Local\node-gyp\Cache\18.12.1',
npm ERR! gyp info spawn args '-Dnode_gyp_dir=C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp',
npm ERR! gyp info spawn args '-Dnode_lib_file=C:\\Users\\USERNAME\\AppData\\Local\\node-gyp\\Cache\\18.12.1\\<(target_arch)\\node.lib',
npm ERR! gyp info spawn args '-Dmodule_root_dir=C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',
npm ERR! gyp info spawn args '--no-parallel',
npm ERR! gyp info spawn args '--generator-output',
npm ERR! gyp info spawn args 'C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build',
npm ERR! gyp info spawn args '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info ok
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp info spawn C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args 'build/binding.sln',
npm ERR! gyp info spawn args '/clp:Verbosity=minimal',
npm ERR! gyp info spawn args '/nologo',
npm ERR! gyp info spawn args '/p:Configuration=Release;Platform=x64'
npm ERR! gyp info spawn args ]
npm ERR! gyp ERR! build error
npm ERR! gyp ERR! stack Error: C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe failed with exit code: 1
npm ERR! gyp ERR! stack at ChildProcess.onExit (C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp\lib\build.js:203:23)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:291:12)
npm ERR! gyp ERR! System Windows_NT 10.0.19044
npm ERR! gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "build" "--fallback-to-build" "--update-binary" "--module=C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build\Release\canvas.node" "--module_name=canvas" "--module_path=C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build\Release" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v108"
npm ERR! gyp ERR! cwd C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas
npm ERR! gyp ERR! node -v v18.12.1
npm ERR! gyp ERR! node-gyp -v v9.3.0
npm ERR! gyp ERR! not ok
npm ERR! node-pre-gyp ERR! build error
npm ERR! node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\USERNAME\AppData\Roaming\nvm\v18.12.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js build --fallback-to-build --update-binary --module=C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build\Release\canvas.node --module_name=canvas --module_path=C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas\build\Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v108' (1)
npm ERR! node-pre-gyp ERR! stack at ChildProcess. (C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules@mapbox\node-pre-gyp\lib\util\compile.js:89:23)
npm ERR! node-pre-gyp ERR! stack at ChildProcess.emit (node:events:513:28)
npm ERR! node-pre-gyp ERR! stack at maybeClose (node:internal/child_process:1091:16)
npm ERR! node-pre-gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm ERR! node-pre-gyp ERR! System Windows_NT 10.0.19044
npm ERR! node-pre-gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\@mapbox\node-pre-gyp\bin\node-pre-gyp" "install" "--fallback-to-build" "--update-binary"
npm ERR! node-pre-gyp ERR! cwd C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client\node_modules\canvas
npm ERR! node-pre-gyp ERR! node -v v18.12.1
npm ERR! node-pre-gyp ERR! node-pre-gyp -v v1.0.10
npm ERR! node-pre-gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\USERNAME\AppData\Local\npm-cache_logs\2023-01-03T14_09_50_142Z-debug-0.log

C:\Users\USERNAME\Programming\FRT_Labelling\Form-Recognizer-Toolkit\SampleCode\LabelingUX\Client>

Unable to start the project

2023-12-12T08_07_33_365Z-debug-0.log
Hi, I was unable to start the project. Showing lot of dependency errors

Resource not found sample_rag_langchain.ipynb

Hi,
I'm trying to run the /SampleCode/Python/sample_rag_langchain.ipynb file. But in the middle when it tries to connect to Azure search it throws the following error.

Does anyone has an idea about fixing this? My Azure Search Instance, OpenAi Instance and Document Intelligence instance are in the same region(East US) and in the same resource group.