RAG工具箱：基于多模态大模型的文档解析方案（2025版）

"https://dashscope-intl.aliyuncs.com/compatible-mode/v1" -> 
"https://dashscope.aliyuncs.com/compatible-mode/v1"

```html
<html><body>
<p>During planning, the Q-planning algorithm randomly samples only from state-action pairs that have previously been experienced (in Step 1), so the model is never queried with a pair about which it has no information.</p> 
 <p>The overall architecture of Dyna agents, of which the Dyna-Q algorithm is one example, is shown in Figure 8.1. The central column represents the basic interaction between agent and environment, giving rise to a trajectory of real experience. The arrow on the left of the figure represents direct reinforcement learning operating on real experience to improve the value function and the policy. On the right are model-based processes. The model is learned from real experience and gives rise to simulated experience. We use the term search control to refer to the process that selects the starting states and actions for the simulated experiences generated by the model. Finally, planning is achieved by applying reinforcement learning methods to the simulated experiences just as if they had really happened. Typically, as in Dyna-Q, the same reinforcement learning method is used both for learning from real experience and for planning from simulated experience. The reinforcement learning method is thus the “final common path” for both learning and planning. Learning and planning are deeply integrated in the sense that they share almost all the same machinery, differing only in the source of their experience.</p> 
 <div class="image"><img/></div> 
 <p>Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between the environment and the policy, affects policy and value functions in much the same way as does simulated experience generated by the model of the environment.</p> 
 <p>Conceptually, planning, acting, model-learning, and direct RL occur simultaneously and in parallel in Dyna agents. For concreteness and implementation on a serial computer, however, we fully specify the order in which they occur within a time step. In Dyna-Q, the acting, model-learning, and direct RL processes require little computation, and we assume they consume just a fraction of the time. The remaining time in each step can be devoted to the planning process, which is inherently computation-intensive. Let us assume that there is time in each step, after acting, model-learning, and direct RL, to complete</p> 
</body></html>
```

pip install mistralai

from mistralai import Mistral

api_key = "API_KEY"
client = Mistral(api_key=api_key)
ocr_model = "mistral-ocr-latest"
system = """You are an AI Assistant with document understanding via URLs. You will be provided with URLs, and you must answer any questions related to those documents.

# OPEN URLS INSTRUCTIONS
You can open URLs by using the `open_urls` tool. It will open webpages and apply OCR to them, retrieving the contents. Use those contents to answer the user.
Only URLs pointing to PDFs and images are supported; you may encounter an error if they are not; provide that information to the user if required."""


def _perform_ocr(url: str) -> str:
    try:   # Apply OCR to the PDF URL
        response = client.ocr.process(
            model=ocr_model,
            document={
                "type": "document_url",
                "document_url": url
                }
            )
    except Exception:
        try:  # IF PDF OCR fails, try Image OCR
            response = client.ocr.process(
                model=ocr_model,
                document={
                    "type": "image_url",
                    "image_url": url
                    }
                )
        except Exception as e:
            return e  # Return the error to the model if it fails, otherwise return the contents
    return "\n\n".join([f"### Page {i+1}\n{response.pages[i].markdown}" for i in range(len(response.pages))])


pdf_response = _perform_ocr("https://arxiv.org/pdf/2201.04234")

{
    "pages": [
        {
            "index": 1,
            "markdown": "markdown-content",
            "images": [
                {
                    "id": "img-0.jpeg",
                    "top_left_x": 292,
                    "top_left_y": 217,
                    "bottom_right_x": 1405,
                    "bottom_right_y": 649,
                    "image_base64": "..."
                }
            ],
            "dimensions": {
                "dpi": 200,
                "height": 2200,
                "width": 1700
            }
        },
      ...
}

from mistralai.models import OCRResponse
from IPython.display import Markdown, display

def replace_images_in_markdown(markdown_str: str, images_dict: dict) -> str:
    for img_name, base64_str in images_dict.items():
        markdown_str = markdown_str.replace(f"![{img_name}]({img_name})", f"![{img_name}]({base64_str})")
    return markdown_str

def get_combined_markdown(ocr_response: OCRResponse) -> str:
  markdowns: list[str] = []
  for page in pdf_response.pages:
    image_data = {}
    for img in page.images:
      image_data[img.id] = img.image_base64
    markdowns.append(replace_images_in_markdown(page.markdown, image_data))

  return "\n\n".join(markdowns)

display(Markdown(get_combined_markdown(pdf_response)))

apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools
git clone https://github.com/allenai/olmocr.git
cd olmocr
pip install -e .
pip install sgl-kernel==0.0.3.post1 --force-reinstall --no-deps
pip install "sglang[all]>=0.4.3.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/

python -m olmocr.pipeline /content/localworkspace --pdfs /content/pdf/*.pdf

{
  "id":"id",
  "text": "markdown-content",
  "source": "olmocr",
  "added": "time",
  "created": "time",
  "metadata": 
    {"Source-File": "file-path", 
      "olmocr-version": "0.1.58", 
      "pdf-total-pages": 17, 
      "total-input-tokens": 67608, 
      "total-output-tokens": 14821, 
      "total-fallback-pages": 0}, 
  "attributes": {"pdf_page_numbers": [[0, 3519, 1], [3519, 8803, 2], [8803, 12566, 3], [12566, 15747, 4], [15747, 19147, 5], [19147, 22975, 6], [22975, 25485, 7], [25485, 31108, 8], [31108, 35902, 9], [35902, 38215, 10], [38215, 39584, 11], [39584, 42844, 12], [42844, 45389, 13], [45389, 48985, 14], [48985, 49297, 15], [49297, 49579, 16], [49579, 51463, 17]]
}

pip install marker-pdf[full]

# 命令行
marker_single /path/to/file.pdf 

from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict
from marker.output import text_from_rendered

converter = PdfConverter(
    artifact_dict=create_model_dict(),
)
rendered = converter("FILEPATH")
text, _, images = text_from_rendered(rendered)

pip install markitdown openai

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("you-pdf-file.pdf")
print(result.text_content)

from markitdown import MarkItDown
from openai import OpenAI

client = OpenAI()
md = MarkItDown(llm_client=client, llm_model="gpt-4o")
result = md.convert("example.jpg")
print(result.text_content)