Implement PEP 621 (pyproject.toml) and migrate to "src" layout

[buhtz]

Please consider to migrate your pyhton project to the last state of the art packaging process using pyproject.toml file only.

I'm aware that this might be a low prio issue currently.

Further reading:

• https://peps.python.org/pep-0621/
• https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

Feel free to ask back if you have any questions.

[buhtz]

For two real world example see

• https://codeberg.org/buhtz/hyperorg
• https://codeberg.org/buhtz/buhtzology

This projects don't have setup.cfg or setup.py file and they do have the "src" layout.

[initialcommit-io]

@buhtz Thx for the suggestion. This is something @paketb0te suggested in #9 (in addition to poetry), which I think you already saw.

I put this on the backburner since I don't have a lot of experience packaging Python projects and wanted to prioritize functionality before overhauling the structure with something I wasn't super familiar with.

Any chance you want to collaborate with @paketb0te on a PR for this?

Otherwise it will probably be a bit before I can brush up on it and do it myself.

[paketb0te]

@buhtz I am not sure I fully understand the advantages of using a src layout, even after reading THIS, THIS and the first half of the lengthy discussion HERE.

To me it seems like the main advantage is that "The src layout helps prevent accidental usage of the in-development copy of the code." - but I am not sure where / how that applies to our use case 🤔

Could you elaborate a bit why you propose the src layout and which advantages it brings?

Please note that I am not against this idea in any way, I just want to understand the reasons for it 😄

[buhtz]

Thanks for asking back. I think I told but I repeat (in my lame English) to be sure you understood.
I'm not a Python Pro and far away from having a deeper understanding of Python internals and design decisions.

btw: You can contact in me in German language if it might help.

My primary reason to use src-layout really is that the "Python Packing Authority" say so. It is the recommended way today.

understand the advantages of using a src layout

Me either. But what I understand from reading some things and in my own experience:

• Most tools expect src layout or they can better handle a Python project repository if it is setup this way.
• It clearer separate the code you want to ship as a release from the rest of your repository with helper scripts and all the other stuff you have.
• With that layout it is also possible to have a "Python Distribution" (the official word for what we often call a package) with multiple "Import Packages". See bit_demo as an example. A "distribution" is what you would upload to PyPi or pack into a deb-file. A import package is what you have with import thepackage. The bit_demo tries to illustrate this. I haven't figured that out myself but had a lot of help from the community.
• I never found myself or read about a disadvantage.

I'm sure that there are much more advantages. But they might be relevant only when you have bigger projects, with a CI pipeline and all that fancy stuff. I always tried to keep it as simple as possible. But my own projects are very tiny and there is no problem with that.

prevent accidental usage of the in-development copy of the code

Might be because of my English but I don't understand the term "in-development copy of the code".

Some short notes about pyproject.toml. I always had "problems" with the package meta data (version string, name, website url, stuff like this). I don't want to declare a version string on two different places. I want everything centralized in the toml file. But not all tools are able to handle that. But I also have IMHO elegant solution for this. In short my package does have a global meta dictionary which is generated from the available package metadata. A real example is my hyperorg project.

meta = _package_metadata_as_dict()
__version__ = meta['Version']
__name__ = meta['Name']

With that solution you can use everything you declared in your toml file in your python code. Not sure if there will come up a better solution from the python folks in the future. I have never shown that to a python pro, e.g. on their mailinglist.

[paketb0te]

@buhtz thanks for the detailed response.
Maybe we can do it step by step, start by migrating from setup.py to pyproject.toml and later change to the src-layout if we find it to be a good tradeoff of simplicity vs. "less room for errors".

[buhtz]

I see no need to separate that steps but I also see no argument against separating them.

I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests".

[initialcommit-io]

Hello again @paketb0te and @buhtz. When packaging the most recent release of git-sim and trying to upload to PyPI I started running into some errors (didn't save them unfortunately) that were due to the way git-sim is packaged. Like you both mentioned looks like pyproject.toml is the new supported way and using the setup.py is basically being deprecated.

I was able to get around this by downgrading setuptools to version 58.2.0, but I think this is a big sign we should migrate the project now so that we can use the most up to date tools going forward.

After reading the details of the "src" layout, it seems to be essentially the recommended way by PyPA, and since we already use editable install for our development process anyway, doesn't seem like there is a downside.

Would either of you like to work on this?

I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests".

FYI the test suite and our unit test stub have already been moved into a tests/ folder.

[buhtz]

Dear initialcommit-io ,
thanks for trying out.

I won't have resource in the next months. But I can help analyzing problems.

Maybe you can try to reproduce your problem and offer your setup in a separate branch .Then I can have a look.

[initialcommit-io]

Thank you. But I have a workaround for the issue for now, so I think we should spend more time actually doing the project.toml and src structure refactorings. @paketb0te Any chance you're interested in helping with this since you had originally mentioned it a few months ago? 😄

[paketb0te]

@initialcommit-io yeah, I can take a stab at it 😃

If there is any bigger work in progress, it would probably be smart to integrate / merge that into the dev branch before restructuring, otherwise I think the (later) merging will be more difficult then neccessary.

[initialcommit-io]

Thanks @paketb0te !!!

Yes the dev branch should be up to date with the latest changes. Mostly its stuff related to the e2e test suite, which could be useful to run for the first time in a real situation after making your changes 😸

[paketb0te]

@initialcommit-io there is from PIL import Image, ImageChops in tests/e2e_tests/utils.py, what package is that?
pillow?

[initialcommit-io]

there is from PIL import Image, ImageChops in tests/e2e_tests/utils.py, what package is that?
pillow?

Yup, should be!

[initialcommit-io]

Closing since the migration to src layout and pyproject.toml was successful! Thanks again @paketb0te for making this happen.