Manual

Init

Initialize a manifest.

usage: dfetch init [-h]

Generate a manifest that can be used as basis for a project.

Dfetch can generate a starting manifest.

It will be created in the current folder.

Check

Check all projects for updates.

usage: dfetch check [-h] [--no-recommendations] [--jenkins-json outfile]
                    [--sarif outfile] [--code-climate outfile]
                    [<project> ...]

Positional Arguments

<project>

Specific project(s) to check

Named Arguments

--no-recommendations, -N

Ignore recommendations from fetched projects.

Default: False

--jenkins-json

Generate a JSON that can be parsed by Jenkins.

--sarif

Generate a Sarif JSON that can be parsed by Github.

--code-climate

Generate a code-climate JSON that can be parsed by Gitlab.

Check all project to see if there are any new updates.

To check if your projects are up-to-date, you can let dfetch check it.

For each project the local version (based on tag or revision) will be compared against the available version. If there are new versions available this will be shown.

Child-manifests

It is possible that fetched projects have manifests of their own. After these projects are fetched (with dfetch update), the manifests are read as well and will be checked to look for further dependencies. If you don’t what recommendations, you can prevent Dfetch checking child-manifests with --no-recommendations.

Reporting

Report issues found during check.

DFetch can report its results of checking in a form that is usable for several other tools. See the respective sections for details about using and configuring those reporters.

All reports can contain the following results:

• unfetched-project

  Project was never fetched. Fetch it using dfetch update.

• up-to-date-project

  Project is up-to-date.

• unavailable-project-version

  Requested project version is unavailable at the remote.

• pinned-but-out-of-date-project

  Project is pinned, but out-of-date. Either ignore this message, or update the version in the manifest.

• out-of-date-project

  Project is out-of-date. Update the project using dfetch update.

• local-changes-in-project

  Project was locally changed. Create a patch file using dfetch diff and add it to your manifest using the patch attribute.

Note

When a dfetch check is performed on a different platform than the original dfetch update the line-endings might result in a false positive of local-changes-in-project.

Jenkins reporter

Dfetch can generate a report that is parseable by Jenkins from the Check results.

Depending on the state of the projects it will create a report with information. If all project are up-to-date, nothing will be added to the report.

The information has several severities:

• high : An unfetched project. Fetch the project to solve the issue.

• normal : An out-of-date project. The project is not pinned and a newer version is available.

• lowa pinned but out-of-date project. The project is pinned to a specific version,

  but a newer version is available.

The report generated is the native json format of the warnings-ng plugin. The plugin will show an overview of the found issues:

When an issues is clicked, you can see the exact location in the manifest where the project is listed.

Usage

Add to pipeline using warnings-ng plugin:

/* For a windows agent */
bat: 'dfetch check --jenkins-json jenkins.json'

/* For a linux agent */
sh: 'dfetch check --jenkins-json jenkins.json'

recordIssues tool: issues(pattern: 'jenkins.json', name: 'DFetch')

With the warnings-ng plugin quality gates thresholds can be set to influence the build result. For example don’t fail when pinned projects are out-of-date.For more information see the quality gate configuration documentation of the warnings-ng plugin.

Sarif reporter

Dfetch can generate a report in the Sarif format that is by Github from the Check results.

Depending on the state of the projects it will create a report with information. If all project are up-to-date, nothing will be added to the report.

DFetch can be listed as part of your github actions during pull requests.

The found results can be inspected in the run. Below an example of a locally changed project.

When clicking on ‘details’ it is possible to see the project in the manifest.

The information has several severities:

• Error : An unfetched project. Fetch the project to solve the issue.

• Warning : An out-of-date project. The project is not pinned and a newer version is available.

• Notea pinned but out-of-date project. The project is pinned to a specific version,

  but a newer version is available.

Usage

A Sarif report can be added to a github action as such:

name: DFetch

on: push

jobs:
dfetch:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Set up python 3.10
      uses: actions/setup-python@v2
      with:
        python-version: "3.10"

    - name: Set up dfetch
      run: |
        python -m pip install --upgrade pip
        pip install dfetch
    - name: Check dependencies
      run: dfetch check --sarif sarif.json

    - name: Upload SARIF file
      uses: github/codeql-action/upload-sarif@v1
      with:
        sarif_file: sarif.json

For more information see the Github Sarif documentation.

Code-climate reporter

Dfetch can generate a report that is parseable by Gitlab/Code-climate from the Check results.

Depending on the state of the projects it will create a report with information. If all project are up-to-date, nothing will be added to the report.

The information has several severities:

• major : An unfetched project. Fetch the project to solve the issue.

• minor : An out-of-date project. The project is not pinned and a newer version is available.

• infoa pinned but out-of-date project. The project is pinned to a specific version,

  but a newer version is available.

The report generated is the code-climate json format, the fields described in the gitlab custom code quality tool documentation are in the report but also all fields listed as required by the code-climate json format.

Gitlab will show the results with the pipeline

And clicking will bring you to the project in the manifest.

In the merge request, gitlab will compare the issues in the current branch with that of the base branch (e.g. master/main). This lets you see if any new issue were introduced or solved.

Usage

Let DFetch perform a check and generate the code-climate json and add the result as artifact in you gitlab-ci runner. See gitlab code quality reports for more information.

dfetch:
  image: "python:3.7"
  script:
  - pip install dfetch
  - dfetch check --code-climate dfetch.json
  artifacts:
    reports:
      codequality: dfetch.json

Report

Generate reports containing information about the projects components.

usage: dfetch report [-h] [-o <filename>] [-t {sbom,list}] [<project> ...]

Positional Arguments

<project>

Specific project(s) to report

Named Arguments

-o, --outfile

Report filename

Default: “report.json”

-t, --type

Possible choices: sbom, list

Type of report to generate.

Default: list

Report can be stdout, sbom

Dfetch can generate multiple reports.

There are several report types that DFetch can generate.

List (default)

Dfetch can generate an report on stdout.

Depending on the state of the projects it will show as much information from the manifest or the metadata (.dfetch_data.yaml).

Software Bill-of-Materials

Dfetch can generate a software Bill-of-Materials (SBOM).

An SBOM lists the components and their supply chain relationships. Downstream users of the software can assess the licenses used and potential risk of dependencies.

The generated SBOM can be used as input for other tools to monitor dependencies. The tools track vulnerabilities or can enforce a license policy within an organization.

See https://cyclonedx.org/use-cases/ for more details.

Dfetch will try to parse the license of project, this is retained during an Update.

Update

Update all modules from the manifest.

usage: dfetch update [-h] [-f] [-N] [<project> ...]

Positional Arguments

<project>

Specific project(s) to update

Named Arguments

-f, --force

Always perform update, ignoring version check or local changes.

Default: False

-N, --no-recommendations

Ignore recommendations from fetched projects.

Default: False

Verifies the manifest and checks all dependencies if updates are available.

Update is the main functionality of dfetch.

You can add Projects to your Manifest and update will fetch the version specified. It tries to determine what kind of vcs it is: git, svn or something else.

Child-manifests

It is possible that fetched projects have manifests of their own. When these projects are fetched (with dfetch update), the manifests are read as well and will be checked to look for further dependencies. If you don’t what recommendations, you can prevent Dfetch checking child-manifests with --no-recommendations.

Validate

Validate a manifest.

usage: dfetch validate [-h]

The Manifest is validated against a schema. See manifest for requirements. Note that each time either update or check is run the manifest is also validated.

Note that you can validate your manifest using Validate.

This will parse your Manifest and check if all fields can be parsed.

Diff

Generate a diff of a project.

usage: dfetch diff [-h] [-r <oldrev>[:<newrev>]] <project>

Positional Arguments

<project>

Project to generate diff from

Named Arguments

-r, --revs

Revision(s) to generate diff from

Default: “”

Create a patch of a project. The diff will be a relative patch file only of the project’s directory.

Dfetch can create a patch file with your local changes to the external project.

If you’ve found any issues with the remote project, you can fix these issues inside the context of your project. To help the upstream project, you can generate a patch file that can be applied by the upstream maintainer. The patch will be generated with the version control system of your main project that contains the manifest.

To generate a patch, Dfetch requires two revisions to determine the changes. You can provide these through the --revs argument.

• If --revs is not provided the changes are calculated between the last revision the metadata file was changed and the working copy (also uncommitted changes).

• If --revs specifies one revision (e.g. --revs 23864ef2), that revision will be used as starting point.

• Alternately both revisions can be explicitly specified, e.g. --revs 23864ef2:4a9cb18.

The below statement will generate a patch for some-project from your manifest.

dfetch diff some-project

Using the generated patch

The patch can be used in the manifest see the Patch attribute for more information. It can also be sent to the upstream maintainer in case of bug fixes.

The patch generated is a relative patch and should be applied specifying the base directory of the git repo. See below for the version control specifics. The patch will also contain content of binary files.

# For git repo's
git apply --verbose --directory='some-project' some-project.patch

# For svn repo's
svn patch some-project.patch

Warning

The path given to --directory when applying the patch in a git repo, must be relative to the base directory of the repo, i.e. the folder where the .git folder is located.

For example if you have the patch Core/MyModule/MySubmodule.patch for files in the directory Core/MyModule/MySubmodule/ and your current working directory is Core/MyModule/. The correct command would be:

git apply --verbose --directory='Core/MyModule/MySubModule` MySubmodule.patch

Freeze

Freeze your projects versions in the manifest as they are on disk.

usage: dfetch freeze [-h]

Generate a new manifest that has all version as they are on disk.

Dfetch can freeze the current versions of the projects.

Say you have the following manifest:

manifest:
    version: 0.0

    projects:
     - name: mymodule
       url: http://git.mycompany.local/mycompany/mymodule

As explained in Revision/Branch/Tag when no version is provided the latest version of the default branch (e.g. trunk, master) of mymodule will be fetched on a DFetch update. When your project becomes stable and you want to rely on a specific version of mymodule you can run dfetch freeze.

First DFetch will rename your old manifest (appended with .backup). After that a new manifest is generated with all the projects as in your original manifest, but each with the specific version as it currently is on disk.

In our above example this would for instance result in:

manifest:
    version: 0.0

    projects:
     - name: mymodule
       url: http://git.mycompany.local/mycompany/mymodule
       tag: v1.0.0

Environment

Get information about the environment dfetch is working in.

usage: dfetch environment [-h]

Generate output that can be used by dfetch developers to investigate issues.

Dfetch can generate output about its working environment.

Import

Generate manifest from existing submodules or externals.

usage: dfetch import [-h]

Look for submodules in a git project or externals in a svn project and create a manifest based on that.

Dfetch can convert your git submodules or svn externals based project to dfetch.

Dfetch will look for all submodules or externals in the current project and generate a manifest with the current versions of the repository.

After importing you will have to remove the submodules or externals and you can let dfetch update by running dfetch update.

Migrating from git submodules

• Make sure your repository is up-to-date.

• Make sure your submodules are up-to-date (git submodules update --init).

• Generate a manifest using dfetch import.

• Remove all git submodules (see How do I remove a submodule ).

• Download all your projects using dfetch update.

• Commit your projects as part of your project.

Switching branches

After importing submodules into a manifest in a branch, you might have some difficulties switching branches. If one branch has submodules in the place were your DFetched project dependencies should be, or other way around. Below both situations, assume a branch feature/use-dfetch with a manifest and master with the original submodules in their place.

Switching from branch with submodules to branch with manifest

When switching from branch with submodules to branch without git will warn that the Dfetched project dependencies will overwrite the submodule that is currently in the same spot.

$ git checkout feature/use-dfetch
error: The following untracked working tree files would be overwritten by checkout:
    MySubmodule/somefile.c
    MySubmodule/someotherfile.c

But git status will show nothing:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

To overcome this, remove the submodule folder and checkout branch feature/use-dfetch:

$ rm -rf MySubmodule
$ git checkout feature/use-dfetch

Switching from branch with manifest to branch with submodules

This situation gives no problem, but the submodules are gone and need to be initialized again. To solve this:

$ git checkout master
$ git submodule update --init
Submodule path 'MySubmodule': checked out '08f95e01b297d8b8c1c9101bde58e75cd4d428ce'

Migrating from SVN externals

• Make sure your repository is up-to-date.

• Generate a manifest using dfetch import.

• Remove all svn externals (see How do I remove svn::externals ).

• Download all your projects using dfetch update.

• Commit your projects as part of your project.