Skip to content

i) .dbf file length check is off by 1.  #464

Closed
2 of 2 issues completed
Closed

Description

@JamesParrott

i)

Run python ./Pyshp/tests/run_benchmarks.py
Reader tests:
Running benchmarks 1 times:
-----------------------------------
                parser |  exec time
-----------------------+-----------
/opt/hostedtoolcache/Python/3.15.2/x64/lib/python3.15/site-packages/shapefile.py:2987: PossiblyCorruptFileHeader: Declared file size in /home/runner/work/IronPyShp/IronPyShp/Pyshp/tests/shapefiles/blockgroups.dbf header: 236774 smaller than actual file size: 236775 (both in bytes)
  warnings.warn(msg, category=PossiblyCorruptFileHeader)
      Read Blockgroups | Running...����������  0.0286 s
             Read Edit | Running...����������   0.012 s
            Read Merge | Running...����������   0.223 s
/opt/hostedtoolcache/Python/3.15.2/x64/lib/python3.15/site-packages/shapefile.py:2987: PossiblyCorruptFileHeader: Declared file size in /home/runner/work/IronPyShp/IronPyShp/PyShp_test_shapefile/ne_10m_admin_1_states_provinces.dbf header: 15161513 smaller than actual file size: 15161514 (both in bytes)
  warnings.warn(msg, category=PossiblyCorruptFileHeader)
      Read States_35MB | Running...����������   0.793 s
/opt/hostedtoolcache/Python/3.15.2/x64/lib/python3.15/site-packages/shapefile.py:2987: PossiblyCorruptFileHeader: Declared file size in /home/runner/work/IronPyShp/IronPyShp/PyShp_test_shapefile/ne_110m_admin_0_tiny_countries.dbf header: 139635 smaller than actual file size: 139636 (both in bytes)
  warnings.warn(msg, category=PossiblyCorruptFileHeader)
   Read Tiny Countries | Running...���������� 0.00745 s
/opt/hostedtoolcache/Python/3.15.2/x64/lib/python3.15/site-packages/shapefile.py:2987: PossiblyCorruptFileHeader: Declared file size in /tmp/tmpmlnacmr9 header: 157486 smaller than actual file size: 157487 (both in bytes)
  warnings.warn(msg, category=PossiblyCorruptFileHeader)
 Read GIS_OSM_zip_10MB | Running...����������   0.181 s

https://github.com/JamesParrott/IronPyShp/actions/runs/43751614365/job/132926656877

ii) Level 5 DOS .dbf files should end in 0x1a, but .dbf files of many shapefiles do not.

Added to run_benchmarks,py:

for file_path in SHAPEFILES.values():
    file_path.read_bytes()
    dbf_file = blockgroups_file.with_suffix(".dbf")
    b = dbf_file.read_bytes()
    last_byte = b[-1:]
    print(f"{last_byte == b"\x1a"}, {last_byte=!r}")
>python run_benchmarks.py
True, last_byte=b'\x1a'
True, last_byte=b'\x1a'
True, last_byte=b'\x1a'
True, last_byte=b'\x1a'
True, last_byte=b'\x1a'
True, last_byte=b'\x1a'

The end of the file is marked by a single byte, with value 0x1A.
https://en.wikipedia.org/wiki/.dbf#File_format_of_Level_5_DOS_dBASE

Activity

  1. changed the title [-]i) .dbf file length check is off by 1. ii) Level 5 DOS .dbf files should end in 0x1a, but .dbf files of many shapefiles do not.[/-] [+]i) .dbf file length check is off by 1. ~~ii) Level 5 DOS .dbf files should end in 0x1a, but .dbf files of many shapefiles do not.~~[/+] on Nov 15, 2026
  2. changed the title [-]i) .dbf file length check is off by 1. ~~ii) Level 5 DOS .dbf files should end in 0x1a, but .dbf files of many shapefiles do not.~~[/-] [+]i) .dbf file length check is off by 1. [/+] on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions