Skip to content

Python binary wheels

Jonathan Lambrechts requested to merge python_binary_wheels into master

This is branch is not ready for merging (but almost). I submit the merge request to open the discussion and get your input.

This is a rewrite of the setup.py file in orther to

  • install the files in standard unix location lib/, include/, bin/, share, ... without the gmsh-xxx-sdk prefix, and the gmsh.py in standard python package location so that the (unreliable) gmsh.pth file is not necessary anymore
  • create binary wheels (i.e. pip python packages) from gitlab-ci to be uploaded un pypi.

Remarks :

  • The maximum size of a wheel allowed on pypi is 60M, the maximum total size of all files uploaded to pypi under "gmsh" is 10G. Currently each wheel is approximately 33M.
  • I keep the "gmsh" bash script wrapper which sets the LD_LIBRARY_PATH (or equivalent on other platforms) before launching the real binary file which is installed in /lib. This may not be necessary, we could assume that if the user can put PREFIX/bin in PATH, he can also put PREFIX/lib in LD_LIBRARY_PATH.
  • I don't know where to put "onelab.py". I would say it should go at the same location as gmsh.py but in the sdk, onelab.py is in /bin/. For now, onelab.py is not included in the wheels.
  • I only tested the linux64 wheel, the other wheels should be tested before pushing them to pypi.
  • For now, the wheels are uploaded to test.pypi.org instead of pypi.org
  • The xxxx-tmp_wheel jobs in the gitlab_ci are copy of the xxx_skd_official_snapshot jobs except that I removed the scp to gmsh.info (and the "only master" filter). Before merging, those jobs should be removed and the sdk_official_snapshot jobs can be used instead (after adding the small artifacts sections).
  • Despite being linked with the static glibc (and co), the libgmsh.so library still depends on the standard shared libraries :
$ ldd gmsh-git-Linux64-sdk/lib/libgmsh.so
	linux-vdso.so.1 (0x00007ffe7ff8b000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007fb53a056000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007fb53a03c000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fb53a036000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fb53a014000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fb539e37000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007fb539cef000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fb539cd5000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007fb539b0c000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007fb53f626000)
  • The manylinux1 tag on the gmsh linux wheels is a plain lie. Pypi will only accept wheels tagged as manylinux1, manylinux2010 and manylinux2014 which correspond to the gcc versions of centos 5, 6, and 7 respectively. But it does not check that the wheels actually satisfies the tag requirements. None of those standards include a graphic stack, there is no way the full gmsh library could fulfill them. The nox version almost satisfies manylinux2014 but it depends on a few libstdc++ symbols with a too recent ABI. My suggestion would be to rebuild the gmsh sdk used too build the linux wheels in a centos 7 docker and to not link it with the static glibc. Then include, both the full and nox library in the python wheel and at runtime try to load the full library and if it fails, fallback to the nox.
  • older pip versions do not recognise the 'manylinux2014' tag. If we decide to use it, on some distributions it will be necessary to upgrade pip (pip install --upgrade pip) before installing gmsh.
  • The whl format does not allow for symlinks, so only the libgmsh-X.Y.so is included in the packages.
  • The setup-whl.py file is simple but not standard. It is not intended to be used outside the gitlab-ci build process.
  • I noticed that the SSH_TOKEN and PYPI_TOKEN variable are not protected, this may constitute a security risk.
  • I noticed that all cmake commands in the gitlab-ci defines a BLAS_LAPACK_LIBRARIES variable which is ignored by cmake.
  • The docker I use to create the wheels comes from another project but that's only because the gmsh docker does not include the setuptools python package.

Merge request reports