How to self-hold and distribute python library dependencies

Posted on In Views: Word count in article: 3k Reading time ≈ 3 mins.

I have multiple projects which share public packages and functions. Before, I just repeat public packages in different projects, and updated them via copy and paste. Management and updating are not convenient this way, as I often forgot which server has updated. So I rebuilt my project and made them use the same public packages hosted on Gitea. Then I just need to update code and deploy them use git. The code is not open source, so I just make the repo private and use deploy key to give read access to servers which run the projects.

How to build a public python libraries

  1. Go to Gitea and create a repo for holding your public packages.

  2. Develop your codes and commits them. The file structure should be as following:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    > tree
    .
    ├── package0
    │   ├── package0
    │   │   ├── __init__.py
    │   │   └── package0.py
    │   └── setup.py
    ├── package1
    │   ├── package1
    │   │   ├── __init__.py
    │   │   └── package1.py
    │   └── setup.py
    └── package2
        ├── package2
        │   ├── __init__.py
        │   └── package2.py
        └── setup.py

    Example for __init__.py:

    1
    from .package0 import Class_name

    Example for setup.py:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    from setuptools import setup, find_packages

    setup(
        name='package0',
        version='1.0.0',
        packages=find_packages(),
        install_requires=[
    	"pymysql",
    	"requests",
    ],
    )

  3. How to test codes locally.

    In you test project, the requirement.txt should like this:

    1
    2
    3
    4
    -e /Users/doublefire_chen/GitHub/PyLibs/package0
    -e /Users/doublefire_chen/GitHub/PyLibs/package1
    -e /Users/doublefire_chen/GitHub/PyLibs/package2
    -e /Users/doublefire_chen/GitHub/PyLibs/package3
    1
    2
    pip install -r requirement.txt
    pip install --upgrade -r requirement.txt # if your public packages have some upgration. For local test, if you update codes in python libraries, you just need to upgrade them use pip without updating version number.

  4. How to distribute code to remote server.

    If your public python library has developed well and can be used in production environment, it is time to distribute your code to servers for running. First of all, make sure your server has read access to git repository. You can make this via setting deploy key on Gitea or make repository public.

    In your production environment, make the requirement.txt like this:

    1
    2
    3
    4
    git+ssh://git@gitea.aka.cy/doublefire.chen/PyLibs.git#egg=package0&subdirectory=package0
    git+ssh://git@gitea.aka.cy/doublefire.chen/PyLibs.git#egg=package1&subdirectory=package1
    git+ssh://git@gitea.aka.cy/doublefire.chen/PyLibs.git#egg=package2&subdirectory=package2
    git+ssh://git@gitea.aka.cy/doublefire.chen/PyLibs.git#egg=package3&subdirectory=package3
    1
    2
    pip install -r requirement.txt
    pip install --upgrade -r requirement.txt # if your public packages have some upgration. For development usage, if you update codes in python libraries, you should update version number before updating by pip cause the installation way is not editable installation like testing locally before.

Reference

  1. https://stackoverflow.com/questions/64436875/how-to-set-up-private-python-project-for-people-to-pip-install
  2. https://stackoverflow.com/questions/57695694/pip-install-editable-package-produces-modulenotfounderror
  3. https://stackoverflow.com/questions/42494229/how-to-pip-install-a-local-python-package/42541684#42541684
  4. https://github.com/psf/requests/blob/main/setup.py
  5. https://www.youtube.com/watch?v=gtyEPynfXY4
  6. https://www.youtube.com/watch?v=r-wwMk5faXo