fedora-llm-docs/Documents/pages/Golang.txt

= Golang Packaging Guidelines
:last-reviewed: 2019-06-19

This document details best practices for packaging Golang packages. Most of it
is automated by an extensive use of macros.

== go2rpm

https://pagure.io/GoSIG/go2rpm/[go2rpm] is tool that automates many of these steps.
It is advisable to try `+go2rpm import_path+` first
before attempting to write a SPEC by hand.

== Import Path

In Golang, packages are referenced by full URLs. Since this URL is
referenced in several places throughout the rpmspec, set the
base import path as a global define at the top of the spec file

[source,RPMSpec]
----
%global goipath     github.com/kr/pretty
----

All macros, including package name, source URL, will be computed from this
value.

[NOTE]
====
*Take the time to identify it accurately.* Changing it later will be
inconvenient.

* it may differ from the repository URL;
* generally, the correct value will be the one used by the project in its
  documentation, coding examples, and build assertions;
* use the gopkg import path for all code states when a project uses it.

If upstream confused itself after multiple forks and renamings, you will need
to fix references to past names in the Go source files, unit tests included.
Perform this fixing in `+%prep+`.
====

== Naming

=== Source packages (src.rpm)

* Golang source packages dedicated to providing code MUST be named after their
  main import path.
  This
  process is automated by the `+%{goname}+` macro.
  This macro will remove any capitalization, "go" keywords, and any duplication in
  the import path.
+
For example:
+
--
** the import path `+github.com/kr/pretty+` will become
   `+golang-github-kr-pretty+`
** the import path `+github.com/DATA-DOG/go-txdb+` will become
   `+golang-github-data-dog-txdb+`
** the import path `+github.com/gopherjs/gopherjs+` will become
   `+golang-github-gopherjs+`
--

+
The filename of spec MUST match the name of the package.

+
If you're not sure what will the name processed by the `+%{goname}+` macro be,
simply build the SRPM with:


 fedpkg --release f31 srpm

+
The SRPM filename will be the one to use in your rpmspec and spec filename.

* Source packages that provide a well-known application such as `+etcd+`
  MUST be named after the application.
  End users do not care about the language their applications are written in. But
  do not name packages after an obscure utility binary that happens to be built by
  the package.


==== Implementation: `+%{gorpmname}+`

`+%gometa+` uses the `+%{gorpmname}+` macro to compute the main `+%{goname}+`
from `+%{goipath}+`.

[NOTE]
====
*`+%{gorpmname}+` can produce collisions*

`+%{gorpmname}+` tries to compute human-friendly and rpm-compatible naming from
Go import paths. It simplifies them, removing redundancies and common
qualifiers.
As a result it is possible for two different import paths to produce the same
result. In that case, feel free to adjust this result manually to avoid the
collision.
====

=== Go code packages: `+%{goname}-devel+`
==== In a source package dedicated to providing Go code

Packages that ship Go code in `+%{goipath}+` should be named
`+%{goname}-devel+`.
If your source package is already named `+%{goname}+` then:

[source,RPMSpec]
----
%package devel
[…]
%description devel
[…]
%files devel -f devel.file-list
----

This has been automated by the `+%{gopkg}+` and `+%{gopdevelkg}+` macros
described in the <<Package metadata>> section below.

==== In a another kind of source package

If your source package is named something other than `+%{goname}+`, you SHOULD
use:

[source,RPMSpec]
----
%package -n %{goname}-devel
[…]
%description -n %{goname}-devel
[…]
%files -n %{goname}-devel -f devel.file-list
----

==== Separate code packages

And, finally, if you wish to split the project Go code in multiple packages,
you can compute the corresponding names with:

[source,RPMSpec]
----
%global goname1 %gorpmname importpath1
[…]
%package -n %{goname1}-devel
[…]
%description -n %{goname1}-devel
[…]
%files -n %{goname1}-devel -f %{goname1}.file-list
----

See also the xref:Golang_advanced.adoc#_dealing_with_cyclic_dependencies[Dealing with cyclic dependencies]
chapter.

Do remember that for Go, each directory is a package. Never separate the .go
files contained in a single directory in different packages.

==== Import path compatibility packages: `+%{compat-%{oldgoname}-devel}+`

When a project can be referenced under multiple import paths, due to forks,
renamings, rehostings, organizational changes, or confusing project
documentation, it is possible to generate compatibility sub-packages to
connect code that uses one of the other import paths to the canonical one.

The canonical import path SHOULD always be the one referenced in the project
documentation.
However some projects do not document import path changes, and rely on HTTPS
redirections (for example https://github.com/docker/docker →
https://github.com/moby/moby).
Such a redirection is a sufficient indicator the canonical import path has
changed (but please make sure with upstream).

The new import path SHOULD be reflected in `+%{goipath}+` and compatibility
import paths MUST be declared with the `+goaltipaths+` macro:

[source,RPMSpec]
----
# A space-separated list of import paths to simulate.
%global goaltipaths
----

For example, the Go library `+github.com/Sirupsen/logrus+` was renamed
`+github.com/sirupsen/logrus+` and the documentation reflects this new import
path.
The packager SHOULD thus request a renaming of his package with a new import
path and a compatibility import path:

[source,RPMSpec]
----
%global goipath     github.com/sirupsen/logrus
%global goaltipaths github.com/Sirupsen/logrus
----

If a project has gone through multiple rename, multiple compatibility import
paths can be specified as well.

[NOTE]
====
*Never defer renamings*

Packagers MUST NOT use import path compatibility sub-packages to alias the
canonical import path to one of the previous namings.
Packagers MUST apply upstream renaming choices to the main `+%{goipath}+` spec
variable and
everything that derives from it, such as `+%{goname}+`.
Deferred renamings introduce friction with upstream and other packagers.
====

=== Go binary packages

The binaries produced by your rpmspec SHOULD generally be listed in the main
package. However, if you want a more appropriate name or split binaries among
different packages, you can create additional binary subpackage.
Of course these package MUST NOT be noarch.

For example we can create the package bbolt that will contain the binary
of the same name:

[source,RPMSpec]
----
%package -n bbolt
[…]
%description -n bbolt
[…]
%files -n bbolt
%license LICENSE
%{_bindir}/bbolt
----

== Versioning

Many Go libraries do not use package versions or have regular releases and
are instead maintained in public version control.
In this case, follow the standard Fedora version conventions. This means that
often Go packages will have a version number of `+0+` and a release number like
`+0.10.20190319git27435c6+`.

Most of this process is automated by macros so that you don't have to specify
the release number yourself.

You first specify either a Version, tag or commit in the header.

[source,RPMSpec]
----
Version:
%global tag
%global commit
----

Then use the `+%gometa+` macro:

 %gometa

The `+%gometa+` macro will automatically process the `+%{?dist}+` tag of the
`+Release+` field to take into account the commit if any.

[NOTE]
====
*Commits vs releases*

You SHOULD package releases in priority.
Please reward the projects that make an effort to identify stable code states.
Only fall back to commits when the project does not release, when the release is
more than six months old, or if
you absolutely need one of the changes of a later commit.
In the later cases please inform the project politely of the reason you needed
to give up on their official releases.
Promoting releases is a way to limit incompatible commit hell.
====

== Go Language Architectures

To compile on various architectures, golang and gcc-go compilers are available.
The golang compiler currently supports x86, x86_64, ppc64le, ppc64 (partially,
see upstream issue#13192), s390x, armv7hl and aarch64.

Binaries SHOULD set ExclusiveArch so that we only attempt to build packages on
those arches.
This is now automatically added by the `+%gometa+` macro by leveraging the
`+%{golang_arches}+` macro.
Packagers can exclude `+%ix86+`
(see https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval[Changes/EncourageI686LeafRemoval])
by passing `+-f+` to the `+%gometa+` macro.
The `+-f+` flag tells `+%gometa+` to set `+ExclusiveArch: %{golang_arches_future}+`
instead of `+ExclusiveArch: %{golang_arches}+`.
`+%{golang_arches_future}+` includes the same architectures as
`+{golang_arches}+` sans `+%ix86+`.

== Dependencies

Packages MUST have `+BuildRequires: go-rpm-macros+`.

This is automated by the `+%gometa+` macro.


=== Automatic Dependency Generation

Most of the `+golang-*+` packages are source code only.
The `+*-devel+` sub-package that includes the source code should explicitly have
provides for the golang imports that it includes.
These provides are automatically deduced from import paths.

Binary builds that include these imports will use them in BuildRequires,
for example:

[source,RPMSpec]
----
BuildRequires: golang(github.com/gorilla/context)
----

===  Bundled or unbundled

At the moment golang projects packaged in Fedora SHOULD be unbundled by
default. It means projects are built from dependencies packaged in Fedora.

For some project it can be reasonable to build from bundled dependencies.
Every bundling needs a proper justification.


=== BuildRequires

The BuildRequires of the project contains the dependencies needed by unit tests
and binaries.

You can gather them manually with `golist`.
For example:[[manual_br]]

[source,bash]
----
 export GOPATH=/home/user/go
 export GO111MODULE=off
 export goipath="github.com/sirupsen/logrus"
 go get $goipath
 (sort -u | xargs -I{} echo "BuildRequires:  golang({})") <<< "$(
  golist --imported --package-path $goipath --skip-self
  golist --imported --package-path $goipath --skip-self --tests
 )"
----

outputs:

[source]
----
BuildRequires:  golang(github.com/stretchr/testify/assert)
BuildRequires:  golang(github.com/stretchr/testify/require)
BuildRequires:  golang(golang.org/x/crypto/ssh/terminal)
----

If automatic buildrequires are available on your build target, you can use
the `+%go_generate_buildrequires+` macro in `+%generate_buildrequires+`:

[source,RPMSpec]
----
%generate_buildrequires
%go_generate_buildrequires
----

This macro leverages `+golist+` to gather build dependencies and tests
dependencies from the package source.

== Testing

You MUST run unit tests.
Due to the nature of Go development, especially the lack of use of semantic
versioning, API breakages are frequent.These need to be detected early and dealt
with upstream.

Some tests may be disabled, especially the following kinds of unit tests are
incompatible with a secure build environment such as mock:

* tests that call a remote server or API over the Internet,
* tests that attempt to reconfigure the system,
* tests that rely on a specific app running on the system, like a database
  or syslog server.

If a test is broken for some other reason, you can disable it the same way.
However, you SHOULD also report the problem upstream. Remember to trace in a
comment why each check was disabled, with links to eventual upstream problem
reports.

== Walkthrough

This chapter will present a typical Go spec file step by step, with comments
and explanations.

=== Spec preamble: `+%{goipath}+`, `+%{forgeurl}+` and `+%gometa+`

==== Usual case

A Go package is identified by its import path. A Go spec file will therefore
start with the `+%{goipath}+` declaration. Don't get it wrong, it will control
the behaviour of the rest of the spec file.

 %global goipath    google.golang.org/api

If your package is hosted on a forge like GitHub, GitLab, Bitbucket or Pagure,
the hosting of the Go package will be automatically deduced from this variable
(typically by prefixing it with \https://). If that is not the case, you need
to declare explicitly the hosting URL with the `+%{forgeurl}+` macro

 %global forgeurl    https://github.com/google/google-api-go-client

The `+%{forgeurl}+` declaration is followed by either `+Version+`, `+commit+` or
`+tag+`. Use the combination that matches your use-case.

 %global commit     2dc3ad4d67ba9a37200c5702d36687a940df1111

[NOTE]
====
You can also define `+date+` to override the mtime of the Source archive.
====

Now that we have all the required variables, the `+%gometa+` macro can be run

 %gometa

It will compute and set the following variables if they are not already set by
the packager:

goname::    an rpm-compatible package name derived from goipath
gosource::  a URL that can be used as SourceX: value
gourl::     a URL that can be used as URL: value

It will delegate processing to the `+%forgemeta+` macro for:

forgesource::       a URL that can be used as SourceX: value
forgesetupargs::    the correct arguments to pass to `+%setup+` for this source
used by `+%forgesetup+` and `+%forgeautosetup+`
archivename::       the source archive filename, without extensions
archiveext::        the source archive filename extensions, without leading dot
archiveurl::        the URL that can be used to download the source archive,
without renaming
topdir::            the source archive top directory (can be empty)
extractdir::        the source directory created inside `+%{_builddir}+` after
using `+%forgesetup+`, `+%forgeautosetup+` or `+%{forgesetupargs}+`
repo::              the repository name
owner::             the repository owner (if used by another computed variable)
shortcommit::       the commit hash clamping used by the forge, if any
scm::               the scm type, when packaging code snapshots: commits or tags
distprefix::        the prefix that needs adding to dist to trace non-release
packaging

Most of the computed variables are both overridable and optional.

Now we can add the remaining elements of the preamble. First, we can define a
multiline description block shared between subpackages:

[source,RPMSpec]
----
%global common_description %{expand:
cmux is a generic Go library to multiplex connections based on their payload.
Using cmux, you can serve gRPC, SSH, HTTPS, HTTP, Go RPC, and pretty much any
other protocol on the same TCP listener.}
----

This description MUST stay within 80 characters per line.

[NOTE]
====
If you need specific devel summary and description, you can also define:

- `+%global godevelsummary+`
- `+%global godeveldescription+`
====

Then we MUST specify the license files that will be added to the devel
subpackages:

`+%global golicenses+`: A space-separated list of shell globs matching the
project license files.

And the possible documentation that SHOULD be included:

`+%global godocs+`:  A space-separated list of shell globs matching the project
documentation files. The Go rpm macros will pick up “.md” files by default
without this.

[NOTE]
====
You can also exclude files from `+%golicense+` and `+%godocs+` with:

- `+%global golicensesex+`
- `+%global godocsex+`

For example you might want to exclude `+INSTALL*+` files from `+%godocs+` as
these should not be provided.
====

=== Source package metadata: `+%{goname}+`, `+%{gourl}+` and `+%{gosource}+`

We can declare the usual rpm headers, using the values computed by `+%gometa+`:

[source,RPMSpec]
----
Name:      %{goname}
# If not set before
Version:
Release:   1%{?dist}
Summary:
License:
URL:       %{gourl}
Source:    %{gosource}
----

You can replace them with manual definitions.
For example, replace `+%{gourl}+` with the project homepage if it exists
separately from the repository URL.
Be careful to only replace `+%{go*}+` variables when it adds value to the
specfile and you understand the consequences. Otherwise you will just add
maintenance-intensive discrepancies in the distribution.

=== BuildRequires

If they are not automatically generated, you can now add the dependencies needed
for package building and unit testings:

[source,RPMSpec]
----
BuildRequires:  golang(github.com/stretchr/testify/assert)
BuildRequires:  golang(github.com/stretchr/testify/require)
BuildRequires:  golang(golang.org/x/crypto/ssh/terminal)
----

See <<manual_br,this section>> on how to get the BuildRequires list manually.

=== Description

Now add the main package description:

[source,RPMSpec]
----
%description
%{common_description}
----

=== Package metadata

The process of declaring Go code packages has been automated with the
`+%{gopkg}+` macro. It will automatically generate a `+%package+` and
`+%description+` for all primary import paths and compatibility ones.

 %gopkg

If you only need to generate Go devel subpackages without compat, use:

 %godevelpkg

If necessary, you can then define one or multiple Go packages that will
contain the binaries being built. See the previous <<Go binary packages>>
section.

=== %prep: `+%goprep+` , removing vendoring

`+%goprep+` unpacks the Go source archives and creates the project “GOPATH” tree
used in the rest of the spec file. It removes vendored (bundled) code:
use the `+-k+` flag if you wish to keep this vendored code, and deal with the
consequences in the rest of the spec.

 %goprep

`+%goprep+` only performs basic vendoring detection. It will miss inventive ways
to vendor code. Remove manually missed vendor code after the `+%goprep+` line.

`+%goprep+` will not fix upstream sources for you. Since all the macro calls
that follow `+%goprep+` assume clean problem-free sources, you need to correct
them just after the `+%goprep+` line:

* replace calls to deprecated import paths with their correct value
* patch code problems
* remove dead code (some upstreams deliberately ship broken source code in
  the hope someone will get around to fix it)

[NOTE]
====
You SHOULD send fixes and problem reports upstream. Any patch used
SHOULD contain a link to an upstream bug report or merge request. At minimum,
a comment SHOULD be added to explain the patch rationale.
====

When you package an import path, that participates in a dependency loop, you
need bootstrapping to manage the initial builds:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
For Go code, that means your bootstrap section should:

* remove unit tests that import other parts of the loop
* remove code that imports other parts of the loop

Sometimes one can resolve dependency loops just by splitting specific
subdirectories in a separate -devel subpackage. See also
xref:Golang_advanced.adoc#_dealing_with_cyclic_dependencies[Dealing with cyclic dependencies].

=== Automatic BuildRequires

If BuildRequires generator are supported, you can now add them to your build:

[source,RPMSpec]
----
%generate_buildrequires
%go_generate_buildrequires
----

=== Packaging a binary: the %build section

If your package is a source package only, you can skip this `+%build+` section
entirely.

Otherwise, you first need to identify manually the project parts that can be
built, and how to name the result.
Practically, it’s any directory containing a main() Go section. Nice projects
put those in `+cmd+` subdirectories named after the command that will be built,
which is what we will document here, but it is not a general rule. Sometimes the
whole `+%goipath+` builds as a single binary.

[source,RPMSpec]
----
for cmd in cmd/* ; do
  %gobuild -o %{gobuilddir}/bin/$(basename $cmd) %{goipath}/$cmd
done
----

=== Installing the packages
==== Source package installation

If you package a library, `+%gopkginstall+` will perform installation steps
for all primary import paths and compatibility ones.

 %gopkginstall

If you only need to install Go devel subpackages without compat, use:

 %godevelinstall

==== Binary package installation

For binaries, we simply create the `+%{_bindir}+` directory in the buildroot
and install the commands as executable in it:

[source,RPMSpec]
----
install -m 0755 -vd                     %{buildroot}%{_bindir}
install -m 0755 -vp %{gobuilddir}/bin/* %{buildroot}%{_bindir}/
----

[NOTE]
====
Be wary of command names which might already exist in Fedora. If you have
any doubt, you can check if the command is already provided in Fedora:

[source,shell]
----
dnf whatprovides --disablerepo="*" --enablerepo=rawhide "/usr/bin/$cmd"
----

If this is the case, use your best judgment to rename the command.
====


=== Running the unit tests: `+%gocheck+`

As said before, you MUST run unit tests in `+%check+`:

 %gocheck

However it is often necessary to disable some of them. You have 3 exclusion
flags to do so:

* `+-d <directory>+`:   exclude the files contained in `+<directory>+`
                        non-recursively (subdirectories are not excluded)
* `+-t <tree root>+`:   exclude the files contained in `+<tree root>+`
                        recursively (subdirectories are excluded)
* `+-r <regexp>+`:      exclude files matching `+<regexp>+`

Remember to document why a test has been disabled.

=== %files declaration
==== When shipping source code

The `+%gopkgfiles+` will process the file list produced in `+%install+` and add
the necessary license and documentation files:

 %gopkgfiles

==== When shipping binaries

Binaries are usually shipped in the main package. This package MUST include
legal files and documentation associated with those binaries.

[source,RPMSpec]
----
%files
%license LICENSE
%doc cmd/foo/README.md
%{_bindir}/*
----

== Examples

=== Simple source package

.golang-github-stretchr-testify.spec
[source,RPMSpec]
----
include::{examplesdir}/golang/golang-github-stretchr-testify.spec[]
----

=== Handling package renames

.golang-github-sirupsen-logrus.spec
[source,RPMSpec]
----
include::{examplesdir}/golang/golang-github-sirupsen-logrus.spec[]
----

=== Simple binary package

.golang-github-boltdb-bolt.spec
[source,RPMSpec]
----
include::{examplesdir}/golang/golang-gopkg-square-jose-2.spec[]
----

== Additional resources

=== Advanced uses cases

** xref:Golang_advanced.adoc#_shipping_additional_files[Shipping additional files]
** xref:Golang_advanced.adoc#_additional_header_declarations[Additional header declarations]
** xref:Golang_advanced.adoc#_dealing_with_cyclic_dependencie[Dealing with cyclic dependencies]

=== Additional annotated templates

** xref:Golang_templates.adoc#_minimal_source_package[Minimal source package]
** xref:Golang_templates.adoc#_full_source_package[Full source package]
** xref:Golang_templates.adoc#_minimal_alternative_import_path[Minimal alternative import path]
** xref:Golang_templates.adoc#_full_alternative_import_path[Full alternative import path]
** xref:Golang_templates.adoc#_minimal_binary[Minimal binary]
** xref:Golang_templates.adoc#_full_binary[Full binary]
** xref:Golang_templates.adoc#_multi_package[Multi package]
** xref:Golang_templates.adoc#_manual_package_deprecated[Manual package (deprecated)]