GithubHelp home page GithubHelp logo

Comments (11)

mockersf avatar mockersf commented on May 26, 2024

@Friz64 any idea on how to fix?

from bevy.

Friz64 avatar Friz64 commented on May 26, 2024

Ouch, sorry about this. No idea why my PR would cause this. I unfortunately can't debug this by myself as I don't have access to appropriate hardware at the moment. Best I can do right now is to do some research, in the hopes that I find something useful.

But what we can always do is to put a #[cfg(not(target_os = "macos"))] over the _window variable in the RawHandleWrapper, so that the PR is essentially negated on macOS. It wouldn't be elegant, but it would work.

from bevy.

BD103 avatar BD103 commented on May 26, 2024

I'm getting this when setting RUST_LOG=debug cargo run --example sprite:

2024-05-04T01:49:22.843807Z  INFO bevy_window::system: No windows are open, exiting
2024-05-04T01:49:22.845024Z  INFO bevy_winit::system: Closing window Entity { index: 0, generation: 1 }
2024-05-04T01:49:22.855889Z DEBUG wgpu_core::present: Created CURRENT Surface Texture Id(22,41,mtl)    
2024-05-04T01:49:22.855926Z DEBUG wgpu_core::device::resource: Create view for texture Id(22,41,mtl) filters usages to TextureUses(COLOR_TARGET)    
2024-05-04T01:49:22.857666Z DEBUG wgpu_core::device::life: Active submission 82 is done    
2024-05-04T01:49:22.857898Z DEBUG present_frames: wgpu_core::present: Removing swapchain texture Id(22,41,mtl) from the device tracker    
2024-05-04T01:49:22.857941Z DEBUG present_frames: wgpu_core::present: Presented. End of Frame
(it hangs from here, with no more info logged)

It looks like it's hanging before it creates the next surface texture after the previous was presented. After checking out the main branch from a week ago, we should additionally have the following logs after Presented. End of Frame before exiting the program:

2024-05-04T01:53:36.461981Z DEBUG wgpu_core::device::life: Active submission 77 is done    
2024-05-04T01:53:36.463780Z DEBUG wgpu_core::resource: Buffer Id(9,1,mtl) map state -> Idle    
2024-05-04T01:53:36.463831Z DEBUG wgpu_core::resource: Buffer Id(4,1,mtl) map state -> Idle    
2024-05-04T01:53:36.463980Z DEBUG wgpu_core::resource: Buffer Id(0,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464853Z DEBUG wgpu_core::resource: Buffer Id(12,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464870Z DEBUG wgpu_core::resource: Buffer Id(13,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464902Z DEBUG wgpu_core::resource: Buffer Id(6,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464924Z DEBUG wgpu_core::resource: Buffer Id(2,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464958Z DEBUG wgpu_core::resource: Buffer Id(7,1,mtl) map state -> Idle    
2024-05-04T01:53:36.464990Z DEBUG wgpu_core::device::life: Active submission 78 is done    
2024-05-04T01:53:36.466244Z DEBUG wgpu_core::resource: Buffer Id(10,1,mtl) map state -> Idle    
2024-05-04T01:53:36.466300Z DEBUG wgpu_core::resource: Buffer Id(8,1,mtl) map state -> Idle
(program exits here)
And RUST_LOG=trace

2024-05-04T01:47:12.377846Z  INFO bevy_window::system: No windows are open, exiting
2024-05-04T01:47:12.379049Z  INFO bevy_winit::system: Closing window Entity { index: 0, generation: 1 }
2024-05-04T01:47:12.389174Z TRACE wgpu_core::storage: User is inserting TextureId(13,90,mtl)    
2024-05-04T01:47:12.389201Z DEBUG wgpu_core::present: Created CURRENT Surface Texture Id(13,90,mtl)    
2024-05-04T01:47:12.389213Z TRACE wgpu_core::track::texture:    tex 13: insert start TextureUses(UNINITIALIZED)    
2024-05-04T01:47:12.389245Z DEBUG wgpu_core::device::resource: Create view for texture Id(13,90,mtl) filters usages to TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.389257Z TRACE wgpu_core::storage: User is inserting TextureViewId(13,90,mtl)    
2024-05-04T01:47:12.389266Z TRACE wgpu_core::device::global: Texture::create_view(Id(13,90,mtl)) -> Id(13,90,mtl)    
2024-05-04T01:47:12.389893Z TRACE wgpu_core::storage: User is inserting StagingBufferId(3,90,mtl)    
2024-05-04T01:47:12.389898Z TRACE wgpu_core::device::queue: Queue::write_buffer Id(7,1,mtl) 16384bytes    
2024-05-04T01:47:12.389914Z TRACE wgpu_core::device::queue: Queue::create_staging_buffer Id(3,90,mtl)    
2024-05-04T01:47:12.389885Z TRACE wgpu_core::device::queue: Queue::write_buffer Id(5,1,mtl) 12bytes    
2024-05-04T01:47:12.389971Z TRACE wgpu_core::storage: User is removing StagingBufferId(3,90,mtl)    
2024-05-04T01:47:12.389979Z TRACE wgpu_core::track::buffer:     buf 7: transition BufferUses(COPY_DST) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390086Z TRACE wgpu_core::storage: User is inserting StagingBufferId(9,90,mtl)    
2024-05-04T01:47:12.390095Z TRACE wgpu_core::device::queue: Queue::create_staging_buffer Id(9,90,mtl)    
2024-05-04T01:47:12.390080Z TRACE wgpu_core::track::buffer:     buf 4: transition BufferUses(UNIFORM) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390127Z TRACE wgpu_core::storage: User is removing StagingBufferId(9,90,mtl)    
2024-05-04T01:47:12.390153Z TRACE wgpu_core::storage: User is inserting StagingBufferId(12,90,mtl)    
2024-05-04T01:47:12.390161Z TRACE wgpu_core::device::queue: Queue::create_staging_buffer Id(12,90,mtl)    
2024-05-04T01:47:12.390159Z TRACE wgpu_core::track::buffer:     buf 5: transition BufferUses(COPY_DST) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390180Z TRACE wgpu_core::storage: User is removing StagingBufferId(12,90,mtl)    
2024-05-04T01:47:12.390189Z TRACE wgpu_core::track::buffer:     buf 8: transition BufferUses(COPY_DST) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390214Z TRACE wgpu_core::track::buffer:     buf 9: transition BufferUses(COPY_DST) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390478Z TRACE wgpu_core::storage: User is inserting BindGroupId(6,90,mtl)    
2024-05-04T01:47:12.390487Z TRACE wgpu_core::device::global: Device::create_bind_group -> Id(6,90,mtl)    
2024-05-04T01:47:12.390492Z TRACE wgpu_core::storage: User is inserting BindGroupId(3,90,mtl)    
2024-05-04T01:47:12.390500Z TRACE wgpu_core::device::global: Device::create_bind_group -> Id(3,90,mtl)    
2024-05-04T01:47:12.390501Z TRACE wgpu_core::device::global: BindGroup::drop Id(11,89,mtl)    
2024-05-04T01:47:12.390514Z TRACE wgpu_core::storage: User is inserting BindGroupId(2,90,mtl)    
2024-05-04T01:47:12.390525Z TRACE wgpu_core::device::global: Device::create_bind_group -> Id(2,90,mtl)    
2024-05-04T01:47:12.390533Z TRACE wgpu_core::device::global: BindGroup::drop Id(10,89,mtl)    
2024-05-04T01:47:12.390545Z TRACE wgpu_core::storage: User is removing BindGroupId(11,89,mtl)    
2024-05-04T01:47:12.390567Z TRACE wgpu_core::storage: User is inserting BindGroupId(4,90,mtl)    
2024-05-04T01:47:12.390572Z TRACE wgpu_core::device::queue: Queue::write_buffer Id(12,1,mtl) 80bytes    
2024-05-04T01:47:12.390577Z TRACE wgpu_core::device::global: Device::create_bind_group -> Id(4,90,mtl)    
2024-05-04T01:47:12.390586Z TRACE wgpu_core::storage: User is removing BindGroupId(10,89,mtl)    
2024-05-04T01:47:12.390603Z TRACE wgpu_core::track::buffer:     buf 12: transition BufferUses(VERTEX) -> BufferUses(COPY_DST)    
2024-05-04T01:47:12.390641Z TRACE wgpu_core::storage: User is inserting BindGroupId(5,90,mtl)    
2024-05-04T01:47:12.390649Z TRACE wgpu_core::device::global: Device::create_bind_group -> Id(5,90,mtl)    
2024-05-04T01:47:12.390660Z TRACE wgpu_core::device::global: BindGroup::drop Id(12,89,mtl)    
2024-05-04T01:47:12.390668Z TRACE wgpu_core::storage: User is removing BindGroupId(12,89,mtl)    
2024-05-04T01:47:12.390870Z TRACE wgpu_core::storage: User is inserting CommandBufferId(0,179,mtl)    
2024-05-04T01:47:12.390880Z TRACE wgpu_core::device::global: Device::create_command_encoder -> Id(0,179,mtl)    
2024-05-04T01:47:12.390940Z TRACE wgpu_core::command::render: Encoding render pass begin in command buffer Id(0,179,mtl)    
2024-05-04T01:47:12.391018Z TRACE wgpu_core::command::render: RenderPass::set_pipeline Id(3,1,mtl)    
2024-05-04T01:47:12.391044Z TRACE wgpu_core::command::render: RenderPass::set_bind_group 0 Id(6,90,mtl)    
2024-05-04T01:47:12.391056Z TRACE wgpu_core::track::buffer:     buf 4: insert BufferUses(UNIFORM)..BufferUses(UNIFORM)    
2024-05-04T01:47:12.391066Z TRACE wgpu_core::command::bind:     Binding [0] = group Id(6,90,mtl)    
2024-05-04T01:47:12.391078Z TRACE wgpu_core::command::render: RenderPass::set_bind_group 1 Id(13,1,mtl)    
2024-05-04T01:47:12.391087Z TRACE wgpu_core::track::texture:    tex 20: insert start TextureUses(RESOURCE)    
2024-05-04T01:47:12.391099Z TRACE wgpu_core::command::bind:     Binding [1] = group Id(13,1,mtl)    
2024-05-04T01:47:12.391108Z TRACE wgpu_core::command::render: RenderPass::set_index_buffer Id(11,1,mtl)    
2024-05-04T01:47:12.391115Z TRACE wgpu_core::track::buffer:     buf 11: insert BufferUses(INDEX)..BufferUses(INDEX)    
2024-05-04T01:47:12.391125Z TRACE wgpu_core::command::render: RenderPass::set_vertex_buffer 0 Id(12,1,mtl)    
2024-05-04T01:47:12.391132Z TRACE wgpu_core::track::buffer:     buf 12: insert BufferUses(VERTEX)..BufferUses(VERTEX)    
2024-05-04T01:47:12.391142Z TRACE wgpu_core::command::render: RenderPass::draw_indexed 6 1 0 0 0    
2024-05-04T01:47:12.391156Z TRACE wgpu_core::command::render: Merging renderpass into cmd_buf Id(0,179,mtl)    
2024-05-04T01:47:12.391165Z TRACE wgpu_core::track::texture:    tex 19: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391173Z TRACE wgpu_core::track::texture:    tex 17: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391195Z TRACE wgpu_core::track::buffer:     buf 4: insert BufferUses(UNIFORM)..BufferUses(UNIFORM)    
2024-05-04T01:47:12.391205Z TRACE wgpu_core::track::buffer:     buf 11: insert BufferUses(INDEX)..BufferUses(INDEX)    
2024-05-04T01:47:12.391213Z TRACE wgpu_core::track::buffer:     buf 12: insert BufferUses(VERTEX)..BufferUses(VERTEX)    
2024-05-04T01:47:12.391223Z TRACE wgpu_core::track::texture:    tex 17: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391231Z TRACE wgpu_core::track::texture:    tex 19: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391238Z TRACE wgpu_core::track::texture:    tex 20: insert start TextureUses(RESOURCE)    
2024-05-04T01:47:12.391338Z TRACE wgpu_core::command::render: Encoding render pass begin in command buffer Id(0,179,mtl)    
2024-05-04T01:47:12.391371Z TRACE wgpu_core::command::render: RenderPass::set_pipeline Id(2,1,mtl)    
2024-05-04T01:47:12.391383Z TRACE wgpu_core::command::render: RenderPass::set_bind_group 0 Id(7,1,mtl)    
2024-05-04T01:47:12.391393Z TRACE wgpu_core::track::texture:    tex 17: insert start TextureUses(RESOURCE)    
2024-05-04T01:47:12.391401Z TRACE wgpu_core::command::bind:     Binding [0] = group Id(7,1,mtl)    
2024-05-04T01:47:12.391410Z TRACE wgpu_core::command::render: RenderPass::draw 3 1 0 0    
2024-05-04T01:47:12.391419Z TRACE wgpu_core::command::render: Merging renderpass into cmd_buf Id(0,179,mtl)    
2024-05-04T01:47:12.391426Z TRACE wgpu_core::track::texture:    tex 13: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391440Z TRACE wgpu_core::track::texture:    tex 13: insert start TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391448Z TRACE wgpu_core::track::texture:    tex 17: transition simple TextureUses(COLOR_TARGET) -> TextureUses(RESOURCE)    
2024-05-04T01:47:12.391474Z TRACE wgpu_core::command: Command buffer Id(0,179,mtl)    
2024-05-04T01:47:12.391500Z TRACE wgpu_core::device::queue: Queue::submit Id(0,1,mtl)    
2024-05-04T01:47:12.391520Z TRACE wgpu_core::track::texture:    tex 13: insert start TextureUses(PRESENT)    
2024-05-04T01:47:12.391541Z TRACE wgpu_core::command: Extracting BakedCommands from CommandBuffer "<CommandBuffer>"    
2024-05-04T01:47:12.391552Z TRACE wgpu_core::device::queue: Stitching command buffer Id(0,179,mtl) before submission    
2024-05-04T01:47:12.391569Z TRACE wgpu_core::track::buffer:     buf 4: transition BufferUses(COPY_DST) -> BufferUses(UNIFORM)    
2024-05-04T01:47:12.391577Z TRACE wgpu_core::track::buffer:     buf 12: transition BufferUses(COPY_DST) -> BufferUses(VERTEX)    
2024-05-04T01:47:12.391586Z TRACE wgpu_core::track::texture:    tex 13: transition simple TextureUses(UNINITIALIZED) -> TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391594Z TRACE wgpu_core::track::texture:    tex 17: transition simple TextureUses(RESOURCE) -> TextureUses(COLOR_TARGET)    
2024-05-04T01:47:12.391609Z TRACE wgpu_core::track::texture:    tex 13: transition simple TextureUses(COLOR_TARGET) -> TextureUses(PRESENT)    
2024-05-04T01:47:12.391624Z TRACE wgpu_core::device::queue: Device after submission 179    
2024-05-04T01:47:12.391679Z DEBUG wgpu_core::device::life: Active submission 178 is done    
2024-05-04T01:47:12.391697Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391723Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391745Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391761Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391770Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391779Z TRACE wgpu_core::resource: Destroy raw StagingBuffer "<StagingBuffer>"    
2024-05-04T01:47:12.391807Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(8,89,mtl)    
2024-05-04T01:47:12.391815Z TRACE wgpu_core::track::stateless: BindGroup Id(8,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391823Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(10,89,mtl)    
2024-05-04T01:47:12.391830Z TRACE wgpu_core::track::stateless: BindGroup Id(10,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391837Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(12,89,mtl)    
2024-05-04T01:47:12.391843Z TRACE wgpu_core::track::stateless: BindGroup Id(12,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391849Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(9,89,mtl)    
2024-05-04T01:47:12.391855Z TRACE wgpu_core::track::stateless: BindGroup Id(9,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391862Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(11,89,mtl)    
2024-05-04T01:47:12.391868Z TRACE wgpu_core::track::stateless: BindGroup Id(11,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391885Z TRACE wgpu_core::binding_model: Destroy raw BindGroup "mesh2d_view_bind_group"    
2024-05-04T01:47:12.391899Z TRACE wgpu_core::binding_model: Destroy raw BindGroup "prepass_view_no_motion_vectors_bind_group"    
2024-05-04T01:47:12.391909Z TRACE wgpu_core::binding_model: Destroy raw BindGroup "ui_view_bind_group"    
2024-05-04T01:47:12.391918Z TRACE wgpu_core::binding_model: Destroy raw BindGroup "mesh2d_view_bind_group"    
2024-05-04T01:47:12.391928Z TRACE wgpu_core::binding_model: Destroy raw BindGroup "sprite_view_bind_group"    
2024-05-04T01:47:12.391940Z TRACE wgpu_core::track::stateless: StatelessTracker::remove_abandoned Id(21,89,mtl)    
2024-05-04T01:47:12.391949Z TRACE wgpu_core::track::stateless: TextureView Id(21,89,mtl) is not tracked anymore    
2024-05-04T01:47:12.391958Z TRACE wgpu_core::resource: Destroy raw TextureView Id(21,89,mtl)    
2024-05-04T01:47:12.391970Z TRACE wgpu_core::track::texture: Texture Id(11,1,mtl) is still referenced from 3    
2024-05-04T01:47:12.391977Z TRACE wgpu_core::resource: Destroy raw Texture "<Surface>"    
2024-05-04T01:47:12.391988Z TRACE wgpu_core::track::buffer: Buffer Id(4,1,mtl) is still referenced from 8    
2024-05-04T01:47:12.391994Z TRACE wgpu_core::track::buffer: Buffer Id(5,1,mtl) is still referenced from 6    
2024-05-04T01:47:12.392059Z DEBUG present_frames: wgpu_core::present: Removing swapchain texture Id(13,90,mtl) from the device tracker    
2024-05-04T01:47:12.392076Z TRACE present_frames: wgpu_core::storage: User is removing TextureId(13,90,mtl)    
2024-05-04T01:47:12.392099Z DEBUG present_frames: wgpu_core::present: Presented. End of Frame    
2024-05-04T01:47:12.392210Z TRACE wgpu_core::device::global: BindGroup::drop Id(3,90,mtl)    
2024-05-04T01:47:12.392220Z TRACE wgpu_core::storage: User is removing BindGroupId(3,90,mtl)    
2024-05-04T01:47:12.392232Z TRACE wgpu_core::device::global: BindGroup::drop Id(5,90,mtl)    
2024-05-04T01:47:12.392239Z TRACE wgpu_core::storage: User is removing BindGroupId(5,90,mtl)

from bevy.

ids1024 avatar ids1024 commented on May 26, 2024

Odd. It seems the Arc in WindowWrapper/RawHandleWrapper does eventually get freed, so the winit::Window is still being dropped, but dropping it later is somehow not working properly?

from bevy.

BD103 avatar BD103 commented on May 26, 2024

I'm a bit new to using a debugger, but it seems to be blocking on a Condvar?

image

It occurs within TaskPool::scope_with_executor_inner:

block_on(async move {

So it appears that something is not waking up the main thread. Perhaps the window Arc is never being dropped because something else made a copy of it, but the event loop depends on it being dropped?

from bevy.

ids1024 avatar ids1024 commented on May 26, 2024

There are probably better ways to debug, but I tested with this:

impl Drop for RawHandleWrapper {
    fn drop(&mut self) {
        println!("Drop RawHandleWrapper: {}", Arc::strong_count(&self._window));
    }
}

And something similar for WindowWrapper.

That shows a drop of a RawHandleWrapper with a ref count of 1. So it seems the winit::Window should be closed at that point.

I'm not familiar enough with macOS APIs to know if there's any macOS-related reason this might happen.

from bevy.

ids1024 avatar ids1024 commented on May 26, 2024

Ah. It looks like the drop impl for the WindowWrapper is called on the main thread, while the drop impl for the RawHandleWrapper is not.

So it's probably an issue that happens when the winit::Window is dropped on a thread other than the main thread.

from bevy.

ids1024 avatar ids1024 commented on May 26, 2024

On macOS, Drop for winit::Window uses https://docs.rs/objc2-foundation/0.2.0/objc2_foundation/struct.MainThreadBound.html Apparently.

On Drop, the inner type is sent to the main thread’s runloop and dropped there. This may lead to deadlocks if the main runloop is not running, or if it is waiting on a lock that the dropping thread is holding. See run_on_main for some of the caveats around that.

from bevy.

ids1024 avatar ids1024 commented on May 26, 2024

Yeah. it looks like the worker thread is hanging in icrate::Foundation::additions::thread::MainThreadMarker::run_on_main. (In lldb, the thread command lets you list and change threads.)

While the main thread is blocked on

block_on(async move {
.

So I guess bevy is blocking the thread running EventLoop::run, which is a problem on platforms like macOS?

from bevy.

Friz64 avatar Friz64 commented on May 26, 2024

Can this now be closed after #13236?

from bevy.

mockersf avatar mockersf commented on May 26, 2024

works fine for me on macOS now πŸ‘

from bevy.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.